Oct 30 17:53:50.134788 ip-10-0-133-62 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 17:53:50.134800 ip-10-0-133-62 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 17:53:50.134809 ip-10-0-133-62 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 17:53:50.135124 ip-10-0-133-62 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 17:54:00.190717 ip-10-0-133-62 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 17:54:00.190733 ip-10-0-133-62 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 14b31835175b4e69b46531c55c3aeb54 -- Oct 30 17:55:45.316943 ip-10-0-133-62 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:55:45.728180 ip-10-0-133-62 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:55:45.728180 ip-10-0-133-62 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:55:45.728180 ip-10-0-133-62 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:55:45.728180 ip-10-0-133-62 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:55:45.732579 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732495 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:55:45.732579 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732576 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:55:45.732579 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732581 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732586 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732610 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732616 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732621 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732625 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732628 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732633 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732637 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732641 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732644 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732648 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732654 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732658 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732662 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732666 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732669 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732673 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732677 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:55:45.732813 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732682 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732686 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732690 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732694 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732698 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732702 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732706 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732710 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732715 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732720 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732724 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732729 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732733 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732737 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732742 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732747 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732751 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732765 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732769 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732774 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:55:45.733630 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732778 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732782 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732786 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732791 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732795 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732802 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732809 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732814 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732819 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732823 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732828 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732832 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732837 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732841 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732846 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732851 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732855 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732860 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732864 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:55:45.734475 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732869 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732873 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732877 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732881 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732885 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732890 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732895 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732902 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732909 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732915 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732919 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732924 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732928 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732932 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732937 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732942 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732946 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732950 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732954 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732959 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:55:45.735336 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732963 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732967 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732972 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732980 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732984 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732988 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732993 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.732998 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733003 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733009 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733013 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733018 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733022 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733027 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733031 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733035 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733039 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733044 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733048 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733052 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:55:45.736287 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733057 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733061 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733066 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733070 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733075 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.733079 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734470 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734482 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734487 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734491 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734495 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734499 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734503 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734508 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734512 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734517 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734522 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734527 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734532 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:55:45.737302 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734537 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734541 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734545 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734549 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734553 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734558 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734563 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734567 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734571 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734576 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734582 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734586 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734613 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734619 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734624 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734629 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734633 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734637 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734642 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:55:45.738198 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734647 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734651 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734655 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734660 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734664 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734668 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734673 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734677 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734681 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734686 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734691 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734696 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734700 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734705 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734709 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734714 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734718 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734722 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734727 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734731 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:55:45.738946 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734735 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734740 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734744 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734750 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734754 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734758 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734763 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734767 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734771 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734775 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734780 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734784 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734788 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734793 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734797 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734801 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734805 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734811 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734815 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734819 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:55:45.739520 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734823 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734828 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734832 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734836 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734841 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734847 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734851 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734856 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734860 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734865 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734869 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734873 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734877 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734881 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734886 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734890 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734895 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734899 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734903 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:55:45.740243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734907 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734912 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734916 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734921 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734925 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734929 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734933 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734938 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734947 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734952 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734956 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734961 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734965 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734969 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.734973 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735814 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735824 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735830 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735835 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:55:45.740786 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735839 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735844 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735848 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735853 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735857 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735861 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735866 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735871 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735875 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735879 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735883 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735888 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735892 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735896 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735901 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735905 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735910 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735914 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735919 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735923 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:55:45.741350 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735927 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735931 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735936 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735943 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735948 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735954 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735960 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735965 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735969 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735973 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735978 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735984 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735992 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.735999 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736004 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736008 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736013 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736017 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736021 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:55:45.741887 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736025 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736030 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736034 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736038 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736043 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736047 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736051 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736055 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736060 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736064 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736068 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736072 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736076 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736080 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736085 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736089 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736098 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736103 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736107 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:55:45.742381 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736111 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736115 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736119 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736124 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736128 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736132 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736138 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736142 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736146 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736150 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736154 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736158 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736163 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736167 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736172 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736176 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736180 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736185 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736189 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736193 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:55:45.742883 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736197 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736201 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736205 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736209 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736214 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736219 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736224 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736228 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736232 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736239 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736243 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736247 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736252 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736256 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736261 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736265 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736269 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736273 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736278 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736282 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:55:45.743392 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736286 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736290 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736294 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.736298 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736433 2573 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736444 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736455 2573 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736462 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736469 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736474 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736481 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736488 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736493 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736498 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736504 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736509 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736514 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736519 2573 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736524 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736529 2573 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736534 2573 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736543 2573 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736549 2573 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:55:45.743919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736554 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736559 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736564 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736570 2573 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736577 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736582 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736587 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736610 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736615 2573 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736621 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736626 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736631 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736635 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736642 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736647 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736652 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736657 2573 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736662 2573 flags.go:64] FLAG: --enable-server="true" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736667 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736676 2573 flags.go:64] FLAG: --event-burst="100" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736681 2573 flags.go:64] FLAG: --event-qps="50" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736686 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736691 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736696 2573 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:55:45.744817 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736702 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736707 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736713 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736718 2573 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736723 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736728 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736738 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736743 2573 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736747 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736751 2573 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736766 2573 flags.go:64] FLAG: --feature-gates="" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736772 2573 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736777 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736782 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736786 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736791 2573 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736795 2573 flags.go:64] FLAG: --help="false" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736800 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-133-62.ec2.internal" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736806 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736811 2573 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736816 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736821 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736826 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736831 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:55:45.745953 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736835 2573 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736839 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736844 2573 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736849 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736854 2573 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736860 2573 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736864 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736869 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736874 2573 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736878 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736882 2573 flags.go:64] FLAG: --lock-file="" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736886 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736891 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736896 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736909 2573 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736914 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736919 2573 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736924 2573 flags.go:64] FLAG: --logging-format="text" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736928 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736934 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736939 2573 flags.go:64] FLAG: --manifest-url="" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736944 2573 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736951 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736956 2573 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736962 2573 flags.go:64] FLAG: --max-pods="110" Oct 30 17:55:45.747043 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736967 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736972 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736978 2573 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736983 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736988 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736993 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.736998 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737009 2573 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737014 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737019 2573 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737024 2573 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737029 2573 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737034 2573 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737039 2573 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737043 2573 flags.go:64] FLAG: --port="10250" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737048 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737053 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fc01e439f55d70eb" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737058 2573 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737063 2573 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737069 2573 flags.go:64] FLAG: --register-node="true" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737073 2573 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737082 2573 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737087 2573 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737092 2573 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737097 2573 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737103 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:55:45.748272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737108 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737113 2573 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737118 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737123 2573 flags.go:64] FLAG: --runonce="false" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737128 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737133 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737139 2573 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737143 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737148 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737155 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737160 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737165 2573 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737170 2573 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737174 2573 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737179 2573 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737184 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737189 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737194 2573 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737198 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737207 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737212 2573 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737216 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737223 2573 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737228 2573 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737233 2573 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:55:45.749491 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737238 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737243 2573 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737252 2573 flags.go:64] FLAG: --v="2" Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737259 2573 flags.go:64] FLAG: --version="false" Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737265 2573 flags.go:64] FLAG: --vmodule="" Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737271 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.737277 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737491 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737498 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737505 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737512 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737520 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737525 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737530 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737535 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737539 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737543 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737548 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737553 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737557 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:55:45.750192 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737562 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737566 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737570 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737575 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737579 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737583 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737587 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737611 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737616 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737620 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737624 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737628 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737635 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737641 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737648 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737652 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737656 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737660 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737664 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:55:45.751064 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737669 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737673 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737677 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737681 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737686 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737690 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737694 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737698 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737702 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737706 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737711 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737716 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737720 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737725 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737729 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737734 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737738 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737742 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737746 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:55:45.751879 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737750 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737754 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737758 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737763 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737768 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737773 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737777 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737781 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737787 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737791 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737795 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737800 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737804 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737808 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737813 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737817 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737821 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737826 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737830 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737834 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:55:45.752358 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737838 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737842 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737847 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737851 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737855 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737861 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737866 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737870 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737874 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737878 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737882 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737887 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737892 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737896 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737900 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737904 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737908 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737913 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737917 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737921 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:55:45.753118 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737927 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737932 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737936 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737941 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737945 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737949 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737953 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737957 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737962 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737966 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737970 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737974 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737978 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737983 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.737987 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.738028 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:55:45.753662 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738083 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738089 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738095 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738099 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738104 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738107 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738111 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738116 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738120 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738127 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738133 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738137 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738142 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738146 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738150 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738154 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738159 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738165 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738169 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:55:45.754100 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738173 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738177 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738182 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738186 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738190 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738194 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738198 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738202 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738206 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738210 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738214 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738218 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738222 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738227 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738233 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738237 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738242 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738247 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738251 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:55:45.754637 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738256 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738261 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738265 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738269 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738273 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738277 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738282 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738286 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738291 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738295 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738299 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738304 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738308 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738312 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738317 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738321 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738327 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738331 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738335 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738340 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:55:45.755141 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738344 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738348 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738352 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738357 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738361 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738365 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738369 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738375 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738381 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738386 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738391 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738395 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738400 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738404 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738408 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738412 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738416 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738421 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738425 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:55:45.755722 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738430 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738434 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738438 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738442 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738446 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738451 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738455 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738459 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738463 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738467 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738471 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738475 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738480 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738484 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738488 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738492 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738499 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738504 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738509 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:55:45.756536 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738514 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738520 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738524 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738528 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738532 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738539 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738543 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738547 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738551 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.738556 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.747147 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.747164 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.747183 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.747190 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747293 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747302 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747307 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747312 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747317 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747321 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:55:45.757379 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747326 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747330 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747334 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747341 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747348 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747353 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747357 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747362 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747366 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747370 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747374 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747378 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747382 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747388 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747393 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747397 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747401 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747407 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747411 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:55:45.758331 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747415 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747419 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747423 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747428 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747432 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747436 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747440 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747445 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747449 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747454 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747458 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747462 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747467 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747471 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747476 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747480 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747484 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747488 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747493 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:55:45.759047 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747497 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747501 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747505 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747509 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747514 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747518 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747523 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747527 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747531 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747535 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747540 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747544 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747549 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747553 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747557 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747562 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747566 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747570 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747574 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747578 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:55:45.759530 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747583 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747588 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747611 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747615 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747619 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747623 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747626 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747630 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747634 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747638 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747642 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747646 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747649 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747654 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747658 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747661 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747665 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747669 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747673 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:55:45.760079 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747677 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747680 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747684 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747688 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747691 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747695 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747700 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747704 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747708 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747712 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747716 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747720 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747724 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747729 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747734 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747738 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747742 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747749 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747754 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:55:45.760585 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747759 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747763 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747768 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.747772 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.747815 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748112 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748120 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748125 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748131 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748137 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748143 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748148 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748154 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748158 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748163 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748167 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:55:45.761243 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748172 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748176 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748180 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748184 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748198 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748203 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748207 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748212 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748216 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748220 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748224 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748228 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748233 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748237 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748242 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748246 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748251 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748255 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748259 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:55:45.761692 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748264 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748268 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748273 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748277 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748281 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748285 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748289 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748293 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748298 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748302 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748306 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748310 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748315 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748320 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748324 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748329 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748333 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748337 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748354 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:55:45.762317 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748360 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748364 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748368 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748372 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748377 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748381 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748385 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748389 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748393 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748398 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748402 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748406 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748410 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748414 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748419 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748425 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748430 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748434 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748438 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748442 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:55:45.762965 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748446 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748450 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748455 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748459 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748463 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748467 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748472 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748476 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748480 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748484 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748489 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748493 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748505 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748510 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748514 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748519 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748525 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748531 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748536 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:55:45.763463 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748541 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748545 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748550 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748554 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748558 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748562 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748566 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748570 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748577 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748581 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748586 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748636 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748642 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748646 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748650 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748654 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748658 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:55:45.763995 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748663 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.748707 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748784 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748794 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748799 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748803 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748808 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748812 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748816 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748828 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748833 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748838 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748842 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748846 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748850 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:55:45.764425 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748855 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748859 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748863 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748867 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748871 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748875 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748880 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748884 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748890 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748894 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748899 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748903 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748907 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748914 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748918 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748923 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748927 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748931 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748935 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748940 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:55:45.764838 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748944 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748948 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748952 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748956 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748960 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748964 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748969 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748983 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748988 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748992 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.748997 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749001 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749005 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749009 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749013 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749018 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749022 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749027 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749031 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:55:45.765339 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749035 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749043 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749049 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749053 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749058 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749062 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749068 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749072 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749077 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749081 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749085 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749089 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749093 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749097 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749102 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749106 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749110 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749115 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749119 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749123 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:55:45.765892 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749128 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749140 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749145 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749149 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749154 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749158 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749165 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749170 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749175 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749179 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749183 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749187 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749191 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749197 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749202 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749206 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749210 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749214 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749220 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749224 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:55:45.766407 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749228 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749232 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749237 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749241 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749245 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749250 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749254 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749259 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749263 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749268 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749272 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749276 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749287 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:45.749291 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.749372 2573 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.752495 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.753636 2573 server.go:1014] "Starting client certificate rotation" Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.753773 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:55:45.766923 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.753816 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:55:45.767397 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.756451 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:55:45.767397 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.760992 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:55:45.769865 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.769852 2573 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:55:45.773891 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.773877 2573 log.go:25] "Validated CRI v1 image API" Oct 30 17:55:45.774764 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.774738 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:55:45.779001 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.778983 2573 fs.go:135] Filesystem UUIDs: map[33969049-38b9-4442-a2f9-c8530e868690:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9bc7bfbd-5451-40d5-a0a7-1721da0f8336:/dev/nvme0n1p4] Oct 30 17:55:45.779082 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.779001 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 17:55:45.785722 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.785618 2573 manager.go:217] Machine: {Timestamp:2026-10-30 17:55:45.784162507 +0000 UTC m=+0.373323906 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3128558 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22dd22e7a4f3bcd61115b1c4bbcb0d SystemUUID:ec22dd22-e7a4-f3bc-d611-15b1c4bbcb0d BootID:14b31835-175b-4e69-b465-31c55c3aeb54 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fd:e4:20:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fd:e4:20:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:3b:b8:3b:1c:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:55:45.785796 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.785724 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:55:45.785796 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.785792 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:55:45.789557 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.789532 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:55:45.789692 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.789559 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-62.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:55:45.790376 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.790362 2573 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:55:45.790410 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.790381 2573 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:55:45.790440 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.790420 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:55:45.790472 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.790439 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:55:45.791947 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.791738 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:55:45.796469 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.796456 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:55:45.796587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.796577 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:55:45.797422 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.797411 2573 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:55:45.797477 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.797425 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:55:45.797477 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.797438 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:55:45.797477 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.797447 2573 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:55:45.797477 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.797454 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:55:45.797783 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.797769 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:55:45.800333 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.800315 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:55:45.806971 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.806959 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:55:45.807951 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.807933 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:55:45.807987 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.807969 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:55:45.808116 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808106 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:55:45.808147 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808123 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:55:45.808147 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808129 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:55:45.808147 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808134 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:55:45.808147 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808140 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:55:45.808147 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808147 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:55:45.808807 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808798 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:55:45.808839 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808808 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:55:45.808839 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808816 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:55:45.808839 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808821 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:55:45.808839 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808830 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:55:45.808839 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.808838 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:55:45.810344 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.810334 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:55:45.810381 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.810346 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:55:45.814055 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.814042 2573 server.go:1263] "Started kubelet" Oct 30 17:55:45.818191 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.817247 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:55:45.818191 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.817359 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:55:45.818191 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.814122 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:55:45.817841 ip-10-0-133-62 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:55:45.820131 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.819028 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:55:45.821532 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.821517 2573 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:55:45.824520 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.824505 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 17:55:45.827915 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.827902 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:55:45.827983 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.827938 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:55:45.828045 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.828028 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:45.829111 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.828685 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:55:45.829111 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.828688 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:55:45.829111 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.828709 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:55:45.829111 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.828927 2573 factory.go:55] Registering systemd factory Oct 30 17:55:45.829111 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.828941 2573 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:55:45.829111 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.829000 2573 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:55:45.829111 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.829011 2573 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:55:45.829454 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.829148 2573 factory.go:153] Registering CRI-O factory Oct 30 17:55:45.829454 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.829163 2573 factory.go:223] Registration of the crio container factory successfully Oct 30 17:55:45.829454 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.829224 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:55:45.829454 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.829243 2573 factory.go:103] Registering Raw factory Oct 30 17:55:45.829454 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.829258 2573 manager.go:1199] Started watching for new ooms in manager Oct 30 17:55:45.830318 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.830293 2573 manager.go:319] Starting recovery of all containers Oct 30 17:55:45.840227 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.840205 2573 manager.go:324] Recovery completed Oct 30 17:55:45.844261 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.844248 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:55:45.845862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.845847 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:55:45.845925 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.845875 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:55:45.845925 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.845885 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:55:45.846387 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.846375 2573 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:55:45.846387 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.846385 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:55:45.846483 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.846400 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:55:45.846483 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.846377 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:55:45.847553 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.847538 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:55:45.847632 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.847565 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:55:45.847632 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.847576 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:55:45.848496 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.848485 2573 policy_none.go:50] "Start" Oct 30 17:55:45.848535 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.848503 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:55:45.848535 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.848517 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:55:45.850497 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.850487 2573 policy_none.go:44] "Start" Oct 30 17:55:45.851952 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.851922 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-62.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:55:45.853264 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.851915 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-62.ec2.internal.18e36049977ce9bd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-62.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-62.ec2.internal,},FirstTimestamp:2026-10-30 17:55:45.814014397 +0000 UTC m=+0.403175792,LastTimestamp:2026-10-30 17:55:45.814014397 +0000 UTC m=+0.403175792,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-62.ec2.internal,}" Oct 30 17:55:45.853392 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.853345 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 17:55:45.853705 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.853685 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:55:45.869858 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.869837 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:55:45.869858 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.869787 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-62.ec2.internal.18e360499962def3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-62.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-62.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-62.ec2.internal,},FirstTimestamp:2026-10-30 17:55:45.845862131 +0000 UTC m=+0.435023526,LastTimestamp:2026-10-30 17:55:45.845862131 +0000 UTC m=+0.435023526,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-62.ec2.internal,}" Oct 30 17:55:45.869976 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.869910 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:55:45.869976 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.869844 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.884711 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mdj5t" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.892815 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.895341 2573 manager.go:341] "Starting Device Plugin manager" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.895389 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.895397 2573 server.go:87] "Starting device plugin registration server" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.895571 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.895579 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.895645 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.895718 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.895725 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.896151 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.896187 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.896386 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mdj5t" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.898423 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.899703 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.899720 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.899735 2573 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.899762 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 17:55:45.906104 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:45.904913 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-62.ec2.internal.18e36049996325b9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-62.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-62.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-62.ec2.internal,},FirstTimestamp:2026-10-30 17:55:45.845880249 +0000 UTC m=+0.435041644,LastTimestamp:2026-10-30 17:55:45.845880249 +0000 UTC m=+0.435041644,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-62.ec2.internal,}" Oct 30 17:55:45.920359 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.920343 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:55:45.996487 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.996444 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:55:45.997163 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.997151 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:55:45.997231 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.997174 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:55:45.997231 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.997184 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:55:45.997231 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:45.997201 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.000314 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.000295 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal"] Oct 30 17:55:46.000355 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.000345 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:55:46.001020 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.001000 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:55:46.001108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.001028 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:55:46.001108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.001042 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:55:46.003259 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.003244 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:55:46.003879 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.003867 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:55:46.003939 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.003887 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:55:46.003939 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.003896 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:55:46.004091 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.004075 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.004126 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.004108 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:55:46.004778 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.004762 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:55:46.004854 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.004794 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:55:46.004854 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.004808 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:55:46.005915 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.005902 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.005971 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.005926 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:55:46.009074 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.009056 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:55:46.009157 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.009085 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:55:46.009157 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.009095 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:55:46.017985 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.017970 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.018055 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.017989 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-62.ec2.internal\": node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:46.029686 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.029664 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.029751 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.029693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.029751 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.029709 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cea7e9c368b2e2c6cdd7c0cacd3b509e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-62.ec2.internal\" (UID: \"cea7e9c368b2e2c6cdd7c0cacd3b509e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.041511 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.041491 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-62.ec2.internal\" not found" node="ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.044235 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.044221 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-62.ec2.internal\" not found" node="ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.070567 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.070552 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:46.130131 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.130110 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.130181 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.130137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.130181 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.130153 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cea7e9c368b2e2c6cdd7c0cacd3b509e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-62.ec2.internal\" (UID: \"cea7e9c368b2e2c6cdd7c0cacd3b509e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.130250 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.130202 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cea7e9c368b2e2c6cdd7c0cacd3b509e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-62.ec2.internal\" (UID: \"cea7e9c368b2e2c6cdd7c0cacd3b509e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.130250 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.130216 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.130311 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.130211 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.170839 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.170823 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:46.270909 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.270862 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:46.346442 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.346421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.349321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.349306 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:55:46.371247 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.371076 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:46.471632 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.471581 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:46.572062 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.572034 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:46.672648 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.672621 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:46.753081 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.753059 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 17:55:46.753468 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.753185 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:55:46.773380 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.773353 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:46.874279 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.874232 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:46.903244 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.903197 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:50:46 +0000 UTC" deadline="2028-04-11 22:41:23.589476413 +0000 UTC" Oct 30 17:55:46.903244 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.903226 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12700h45m36.686254036s" Oct 30 17:55:46.975271 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:46.975253 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:55:46.976326 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:46.976312 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:55:47.005114 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:47.005089 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea7e9c368b2e2c6cdd7c0cacd3b509e.slice/crio-8ce825231d61103e72d15ad82becc935dfe0cbea92fe490337dd2ad2ce9c25ac WatchSource:0}: Error finding container 8ce825231d61103e72d15ad82becc935dfe0cbea92fe490337dd2ad2ce9c25ac: Status 404 returned error can't find the container with id 8ce825231d61103e72d15ad82becc935dfe0cbea92fe490337dd2ad2ce9c25ac Oct 30 17:55:47.005328 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:47.005315 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8da8176727de6d4893cff0cfaade9af.slice/crio-ef7cbda66828c3a9685d9e4499fa9ba1f503c8e504781bde67511421036afe0d WatchSource:0}: Error finding container ef7cbda66828c3a9685d9e4499fa9ba1f503c8e504781bde67511421036afe0d: Status 404 returned error can't find the container with id ef7cbda66828c3a9685d9e4499fa9ba1f503c8e504781bde67511421036afe0d Oct 30 17:55:47.011123 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.011107 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:55:47.029124 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.029100 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:55:47.055745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.055728 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:55:47.057792 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.057780 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:55:47.077390 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.077369 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:55:47.206752 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.206698 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:55:47.377281 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.377258 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:55:47.799628 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.799587 2573 apiserver.go:52] "Watching apiserver" Oct 30 17:55:47.829346 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.829327 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 17:55:47.835017 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.834316 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:55:47.835017 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.834631 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-lzl5g","openshift-network-operator/iptables-alerter-t92pz","kube-system/global-pull-secret-syncer-2pp59","openshift-cluster-node-tuning-operator/tuned-nmtmw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal","openshift-multus/multus-additional-cni-plugins-bg76c","openshift-multus/multus-f5mwt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj","openshift-ovn-kubernetes/ovnkube-node-ddxgl","kube-system/konnectivity-agent-wv5gn","openshift-dns/node-resolver-94ttz","openshift-image-registry/node-ca-l67qw","kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal","openshift-multus/network-metrics-daemon-z4mc4"] Oct 30 17:55:47.844095 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.837449 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.844095 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.839808 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:47.844095 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.842232 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:47.844095 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:47.842330 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:55:47.845023 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.845001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.847139 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.847117 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:47.847242 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:47.847198 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:55:47.849945 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.849869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:47.854056 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.852258 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.855099 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.855005 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:47.855099 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:47.855054 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:55:47.856419 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.856396 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8vdz2\"" Oct 30 17:55:47.856521 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.856507 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:55:47.856603 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.856590 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:55:47.856772 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.856758 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:55:47.856823 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.856804 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:55:47.856874 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.856861 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:55:47.858361 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.858337 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:55:47.862599 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.861541 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:55:47.862599 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.861917 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:55:47.862599 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.861959 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:55:47.862599 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.861964 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:55:47.862599 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.861975 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z9xqn\"" Oct 30 17:55:47.862599 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.862189 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pc6fd\"" Oct 30 17:55:47.864382 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.864077 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-94ttz" Oct 30 17:55:47.867438 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.867418 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:55:47.894354 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.894331 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:55:47.894447 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.894408 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9kl5l" Oct 30 17:55:47.906000 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.905952 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" event={"ID":"f8da8176727de6d4893cff0cfaade9af","Type":"ContainerStarted","Data":"ef7cbda66828c3a9685d9e4499fa9ba1f503c8e504781bde67511421036afe0d"} Oct 30 17:55:47.907443 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.907413 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" event={"ID":"cea7e9c368b2e2c6cdd7c0cacd3b509e","Type":"ContainerStarted","Data":"8ce825231d61103e72d15ad82becc935dfe0cbea92fe490337dd2ad2ce9c25ac"} Oct 30 17:55:47.918275 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.918257 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:55:47.930073 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.930052 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:55:47.939391 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-conf-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.939498 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cnibin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.939498 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.939498 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-cnibin\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:47.939702 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939503 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-systemd-units\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.939702 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939546 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-log-socket\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.939702 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-system-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.939702 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-kubelet\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.939702 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939677 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j827\" (UniqueName: \"kubernetes.io/projected/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-kube-api-access-9j827\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.939963 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939715 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-socket-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:47.939963 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939740 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-tuned\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.939963 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939775 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp6n4\" (UniqueName: \"kubernetes.io/projected/e882582b-e97c-4171-8cff-eba17371f519-kube-api-access-pp6n4\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.939963 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-multus-certs\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.939963 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:47.939963 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-etc-selinux\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:47.939963 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.939978 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-script-lib\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940061 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cni-binary-copy\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn8dq\" (UniqueName: \"kubernetes.io/projected/f1043899-80a3-47a9-9d1c-0421cd68df00-kube-api-access-pn8dq\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-registration-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940146 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-sys\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-tmp\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-slash\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940242 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-node-log\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-config\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940292 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/167ba867-3860-40d7-9d55-9c81bf49930b-agent-certs\") pod \"konnectivity-agent-wv5gn\" (UID: \"167ba867-3860-40d7-9d55-9c81bf49930b\") " pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:55:47.940321 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940308 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4572363-4113-4e99-a34a-1b33a4cc08ff-host\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-sys-fs\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940352 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxjhv\" (UniqueName: \"kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv\") pod \"network-check-target-lzl5g\" (UID: \"92ba2348-2452-4bd2-92ba-a980fdc8a15f\") " pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-ovn\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940397 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-bin\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-kubernetes\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940466 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mdxv\" (UniqueName: \"kubernetes.io/projected/6dc28a5a-54a6-4530-b383-4156404c6f4a-kube-api-access-8mdxv\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e882582b-e97c-4171-8cff-eba17371f519-ovn-node-metrics-cert\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940507 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-bin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940541 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-dbus\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940568 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1043899-80a3-47a9-9d1c-0421cd68df00-hosts-file\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-conf\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:47.940966 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8qp2\" (UniqueName: \"kubernetes.io/projected/c3bd5eaa-c913-4709-8632-5abec1047a5e-kube-api-access-f8qp2\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940802 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-k8s-cni-cncf-io\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln9gq\" (UniqueName: \"kubernetes.io/projected/13295b32-a6cc-4918-bfce-9e7819398cb0-kube-api-access-ln9gq\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940934 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-systemd\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.940993 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-run\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941017 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-host\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941041 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-system-cni-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941062 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-netns\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941087 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941112 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3bd5eaa-c913-4709-8632-5abec1047a5e-host-slash\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-hostroot\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941148 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-kubelet-config\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4572363-4113-4e99-a34a-1b33a4cc08ff-serviceca\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3bd5eaa-c913-4709-8632-5abec1047a5e-iptables-alerter-script\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941209 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-etc-kubernetes\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941232 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/167ba867-3860-40d7-9d55-9c81bf49930b-konnectivity-ca\") pod \"konnectivity-agent-wv5gn\" (UID: \"167ba867-3860-40d7-9d55-9c81bf49930b\") " pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:55:47.941587 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941257 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1043899-80a3-47a9-9d1c-0421cd68df00-tmp-dir\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941282 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-systemd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941344 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-multus\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941359 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-device-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-lib-modules\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941396 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-var-lib-kubelet\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941412 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vq4f\" (UniqueName: \"kubernetes.io/projected/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-kube-api-access-7vq4f\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22rll\" (UniqueName: \"kubernetes.io/projected/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-kube-api-access-22rll\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941457 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-env-overrides\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-netns\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-modprobe-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-var-lib-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941561 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-etc-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941579 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-netd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.942297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-daemon-config\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.942931 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:47.942931 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94n76\" (UniqueName: \"kubernetes.io/projected/f4572363-4113-4e99-a34a-1b33a4cc08ff-kube-api-access-94n76\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:55:47.942931 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysconfig\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:47.942931 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-kubelet\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:47.942931 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.941702 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-socket-dir-parent\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:47.965018 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.964993 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:55:47.965130 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965115 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:55:47.965187 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965124 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:55:47.965745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965362 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:55:47.965745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965449 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:55:47.965745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965538 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:55:47.965745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965553 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:55:47.965745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965585 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:55:47.965745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965611 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:55:47.965745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965644 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:55:47.965745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965645 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:55:47.965745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965668 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:55:47.965745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.965700 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:55:47.977176 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.977079 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:55:47.977176 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.977090 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vkdz6\"" Oct 30 17:55:47.977176 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.977099 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:55:47.977176 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.977140 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:55:47.979847 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.979815 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:55:47.980575 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.980551 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:55:47.980791 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.980773 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z6vc6\"" Oct 30 17:55:47.980861 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.980795 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mmrm7\"" Oct 30 17:55:47.980861 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.980816 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zxlb9\"" Oct 30 17:55:47.980861 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.980777 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-26whl\"" Oct 30 17:55:47.981012 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.980887 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v7j5r\"" Oct 30 17:55:47.981012 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:47.980994 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:55:48.041964 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.041941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-cnibin\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.042093 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-systemd-units\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.042093 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-log-socket\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.042093 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042017 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-cnibin\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.042093 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-system-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.042308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042109 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-systemd-units\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.042308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-kubelet\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.042308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042115 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-system-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.042308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042111 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-log-socket\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.042308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j827\" (UniqueName: \"kubernetes.io/projected/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-kube-api-access-9j827\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.042308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-socket-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.042308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-tuned\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.042308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042188 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-kubelet\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.042308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp6n4\" (UniqueName: \"kubernetes.io/projected/e882582b-e97c-4171-8cff-eba17371f519-kube-api-access-pp6n4\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.042308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-multus-certs\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042346 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-multus-certs\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042362 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-socket-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-etc-selinux\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042549 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-script-lib\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042611 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042595 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cni-binary-copy\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8dq\" (UniqueName: \"kubernetes.io/projected/f1043899-80a3-47a9-9d1c-0421cd68df00-kube-api-access-pn8dq\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042710 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-registration-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-sys\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.042795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042774 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-tmp\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-slash\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042831 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-node-log\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-config\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/167ba867-3860-40d7-9d55-9c81bf49930b-agent-certs\") pod \"konnectivity-agent-wv5gn\" (UID: \"167ba867-3860-40d7-9d55-9c81bf49930b\") " pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4572363-4113-4e99-a34a-1b33a4cc08ff-host\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-sys-fs\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjhv\" (UniqueName: \"kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv\") pod \"network-check-target-lzl5g\" (UID: \"92ba2348-2452-4bd2-92ba-a980fdc8a15f\") " pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-ovn\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-bin\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-kubernetes\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdxv\" (UniqueName: \"kubernetes.io/projected/6dc28a5a-54a6-4530-b383-4156404c6f4a-kube-api-access-8mdxv\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043146 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e882582b-e97c-4171-8cff-eba17371f519-ovn-node-metrics-cert\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043173 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-bin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.043229 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043195 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043237 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4572363-4113-4e99-a34a-1b33a4cc08ff-host\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043264 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-script-lib\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043300 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-sys-fs\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-dbus\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1043899-80a3-47a9-9d1c-0421cd68df00-hosts-file\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-conf\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043415 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-registration-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.042644 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-etc-selinux\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043485 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-node-log\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8qp2\" (UniqueName: \"kubernetes.io/projected/c3bd5eaa-c913-4709-8632-5abec1047a5e-kube-api-access-f8qp2\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043549 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-k8s-cni-cncf-io\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln9gq\" (UniqueName: \"kubernetes.io/projected/13295b32-a6cc-4918-bfce-9e7819398cb0-kube-api-access-ln9gq\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-systemd\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043622 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-run\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.043717 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-host\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043700 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-system-cni-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-netns\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043789 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3bd5eaa-c913-4709-8632-5abec1047a5e-host-slash\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-hostroot\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043834 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-kubelet-config\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043856 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4572363-4113-4e99-a34a-1b33a4cc08ff-serviceca\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3bd5eaa-c913-4709-8632-5abec1047a5e-iptables-alerter-script\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-etc-kubernetes\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/167ba867-3860-40d7-9d55-9c81bf49930b-konnectivity-ca\") pod \"konnectivity-agent-wv5gn\" (UID: \"167ba867-3860-40d7-9d55-9c81bf49930b\") " pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1043899-80a3-47a9-9d1c-0421cd68df00-tmp-dir\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043976 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-systemd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043995 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-config\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043995 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-multus\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.044364 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-device-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044081 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-lib-modules\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-var-lib-kubelet\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044171 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vq4f\" (UniqueName: \"kubernetes.io/projected/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-kube-api-access-7vq4f\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044199 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-lib-modules\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rll\" (UniqueName: \"kubernetes.io/projected/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-kube-api-access-22rll\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044256 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3bd5eaa-c913-4709-8632-5abec1047a5e-host-slash\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044311 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-hostroot\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044309 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044358 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-bin\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-env-overrides\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044425 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-kubernetes\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-netns\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044473 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-netns\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044571 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-modprobe-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044631 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-kubelet-config\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:48.044862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044637 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-var-lib-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-etc-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-netd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-daemon-config\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044774 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044797 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-env-overrides\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044802 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n76\" (UniqueName: \"kubernetes.io/projected/f4572363-4113-4e99-a34a-1b33a4cc08ff-kube-api-access-94n76\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044827 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysconfig\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-kubelet\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-socket-dir-parent\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-conf-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cnibin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044976 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.045113 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4572363-4113-4e99-a34a-1b33a4cc08ff-serviceca\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.045134 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.045195 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-etc-kubernetes\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.045646 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-sys\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.045723 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/167ba867-3860-40d7-9d55-9c81bf49930b-konnectivity-ca\") pod \"konnectivity-agent-wv5gn\" (UID: \"167ba867-3860-40d7-9d55-9c81bf49930b\") " pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:55:48.045737 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.045733 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-system-cni-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046007 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1043899-80a3-47a9-9d1c-0421cd68df00-tmp-dir\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046097 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-systemd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046157 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046174 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-k8s-cni-cncf-io\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046204 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-multus\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046252 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-device-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046427 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-netns\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046450 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-tuned\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046485 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-var-lib-kubelet\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046499 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-systemd\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046529 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-etc-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046603 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-run\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.046619 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:55:48.046691 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046653 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-host\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.046739 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs podName:6fe0d1a0-3adb-4b33-89b6-1f39fb282399 nodeName:}" failed. No retries permitted until 2026-10-30 17:55:48.546701579 +0000 UTC m=+3.135862977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs") pod "network-metrics-daemon-z4mc4" (UID: "6fe0d1a0-3adb-4b33-89b6-1f39fb282399") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046742 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-netd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.046793 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.046847 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret podName:e718a01a-548e-4499-95dc-7cbbd6ad6a12 nodeName:}" failed. No retries permitted until 2026-10-30 17:55:48.546824995 +0000 UTC m=+3.135986393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret") pod "global-pull-secret-syncer-2pp59" (UID: "e718a01a-548e-4499-95dc-7cbbd6ad6a12") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046791 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-bin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046906 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-dbus\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.046972 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1043899-80a3-47a9-9d1c-0421cd68df00-hosts-file\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043358 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cni-binary-copy\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.047036 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysconfig\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.047096 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-conf\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.047130 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-modprobe-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.047137 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-var-lib-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.043447 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-slash\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.047371 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.047326 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-socket-dir-parent\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.048081 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.047383 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-conf-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.048081 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.047337 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cnibin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.048081 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.047459 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.048081 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.044317 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-ovn\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.048081 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.047332 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-kubelet\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.048350 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.048184 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e882582b-e97c-4171-8cff-eba17371f519-ovn-node-metrics-cert\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.048350 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.048296 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-daemon-config\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.048488 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.048454 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-tmp\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.048789 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.048756 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.048894 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.048763 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/167ba867-3860-40d7-9d55-9c81bf49930b-agent-certs\") pod \"konnectivity-agent-wv5gn\" (UID: \"167ba867-3860-40d7-9d55-9c81bf49930b\") " pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:55:48.049425 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.049384 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3bd5eaa-c913-4709-8632-5abec1047a5e-iptables-alerter-script\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:55:48.069105 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.069062 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8dq\" (UniqueName: \"kubernetes.io/projected/f1043899-80a3-47a9-9d1c-0421cd68df00-kube-api-access-pn8dq\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:55:48.069284 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.069263 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j827\" (UniqueName: \"kubernetes.io/projected/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-kube-api-access-9j827\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.070150 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.070132 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:55:48.070244 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.070167 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:55:48.070244 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.070181 2573 projected.go:196] Error preparing data for projected volume kube-api-access-zxjhv for pod openshift-network-diagnostics/network-check-target-lzl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:55:48.070335 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.070254 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv podName:92ba2348-2452-4bd2-92ba-a980fdc8a15f nodeName:}" failed. No retries permitted until 2026-10-30 17:55:48.570236533 +0000 UTC m=+3.159397935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zxjhv" (UniqueName: "kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv") pod "network-check-target-lzl5g" (UID: "92ba2348-2452-4bd2-92ba-a980fdc8a15f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:55:48.075281 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.074871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94n76\" (UniqueName: \"kubernetes.io/projected/f4572363-4113-4e99-a34a-1b33a4cc08ff-kube-api-access-94n76\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:55:48.075281 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.074958 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln9gq\" (UniqueName: \"kubernetes.io/projected/13295b32-a6cc-4918-bfce-9e7819398cb0-kube-api-access-ln9gq\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.080990 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.080963 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp6n4\" (UniqueName: \"kubernetes.io/projected/e882582b-e97c-4171-8cff-eba17371f519-kube-api-access-pp6n4\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.082280 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.082235 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vq4f\" (UniqueName: \"kubernetes.io/projected/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-kube-api-access-7vq4f\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.083428 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.083404 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8qp2\" (UniqueName: \"kubernetes.io/projected/c3bd5eaa-c913-4709-8632-5abec1047a5e-kube-api-access-f8qp2\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:55:48.083609 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.083559 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rll\" (UniqueName: \"kubernetes.io/projected/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-kube-api-access-22rll\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:48.085067 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.085041 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdxv\" (UniqueName: \"kubernetes.io/projected/6dc28a5a-54a6-4530-b383-4156404c6f4a-kube-api-access-8mdxv\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.159909 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.159837 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f5mwt" Oct 30 17:55:48.170799 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:48.170768 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ef705c_0b43_46a9_82ba_fe9a8d426ff3.slice/crio-2e84fbad55f885a2bb894c95b6d4b4280cd7e6dadb02873fe5aa6d4d447e6201 WatchSource:0}: Error finding container 2e84fbad55f885a2bb894c95b6d4b4280cd7e6dadb02873fe5aa6d4d447e6201: Status 404 returned error can't find the container with id 2e84fbad55f885a2bb894c95b6d4b4280cd7e6dadb02873fe5aa6d4d447e6201 Oct 30 17:55:48.173620 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.173594 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:55:48.180055 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.179842 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:55:48.183940 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:48.183918 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13295b32_a6cc_4918_bfce_9e7819398cb0.slice/crio-12e41e5e05545f170de90c819a3692a0a7b2a6bf52372ecc4a7c0e7b7da34845 WatchSource:0}: Error finding container 12e41e5e05545f170de90c819a3692a0a7b2a6bf52372ecc4a7c0e7b7da34845: Status 404 returned error can't find the container with id 12e41e5e05545f170de90c819a3692a0a7b2a6bf52372ecc4a7c0e7b7da34845 Oct 30 17:55:48.187873 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.187381 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:55:48.190942 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:48.190917 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd43387_11ba_4288_a0f3_5a16a7d3ce7d.slice/crio-33026e27915ca6ff982c8be6a7c493da5256310a700f13c21dd8208c396f5309 WatchSource:0}: Error finding container 33026e27915ca6ff982c8be6a7c493da5256310a700f13c21dd8208c396f5309: Status 404 returned error can't find the container with id 33026e27915ca6ff982c8be6a7c493da5256310a700f13c21dd8208c396f5309 Oct 30 17:55:48.217118 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.217096 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:55:48.219283 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.219261 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:55:48.229836 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.229814 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:55:48.230459 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:48.230424 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc28a5a_54a6_4530_b383_4156404c6f4a.slice/crio-9d02664274cfb9bf9de77c00f6954e5b228ca3ba9ff4e19497d60a0c04a14a91 WatchSource:0}: Error finding container 9d02664274cfb9bf9de77c00f6954e5b228ca3ba9ff4e19497d60a0c04a14a91: Status 404 returned error can't find the container with id 9d02664274cfb9bf9de77c00f6954e5b228ca3ba9ff4e19497d60a0c04a14a91 Oct 30 17:55:48.231819 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.231770 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-94ttz" Oct 30 17:55:48.231819 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:48.231789 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3bd5eaa_c913_4709_8632_5abec1047a5e.slice/crio-b1f52a68261f62e38dd998161ceb6188a65e071c483afa4a3b43b1eb4f3ed105 WatchSource:0}: Error finding container b1f52a68261f62e38dd998161ceb6188a65e071c483afa4a3b43b1eb4f3ed105: Status 404 returned error can't find the container with id b1f52a68261f62e38dd998161ceb6188a65e071c483afa4a3b43b1eb4f3ed105 Oct 30 17:55:48.239916 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.239886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:55:48.245708 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:48.245677 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167ba867_3860_40d7_9d55_9c81bf49930b.slice/crio-80ed2bfab374e0140e9c9253103658cb87017bfb519929d673b82c54d75efd4e WatchSource:0}: Error finding container 80ed2bfab374e0140e9c9253103658cb87017bfb519929d673b82c54d75efd4e: Status 404 returned error can't find the container with id 80ed2bfab374e0140e9c9253103658cb87017bfb519929d673b82c54d75efd4e Oct 30 17:55:48.247496 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:48.247440 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1043899_80a3_47a9_9d1c_0421cd68df00.slice/crio-9e2650d8ec1d7c969cf0fee983ebe7c48d0543299c11bcf27c6b7fe631b9ad15 WatchSource:0}: Error finding container 9e2650d8ec1d7c969cf0fee983ebe7c48d0543299c11bcf27c6b7fe631b9ad15: Status 404 returned error can't find the container with id 9e2650d8ec1d7c969cf0fee983ebe7c48d0543299c11bcf27c6b7fe631b9ad15 Oct 30 17:55:48.254173 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:55:48.254146 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode882582b_e97c_4171_8cff_eba17371f519.slice/crio-0c15a7e453d497f6a249ff157b6b3a54ba12fa006933cca34c9fe902edc35786 WatchSource:0}: Error finding container 0c15a7e453d497f6a249ff157b6b3a54ba12fa006933cca34c9fe902edc35786: Status 404 returned error can't find the container with id 0c15a7e453d497f6a249ff157b6b3a54ba12fa006933cca34c9fe902edc35786 Oct 30 17:55:48.548864 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.548782 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:48.548864 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.548855 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:48.549120 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.548981 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:55:48.549120 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.549039 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs podName:6fe0d1a0-3adb-4b33-89b6-1f39fb282399 nodeName:}" failed. No retries permitted until 2026-10-30 17:55:49.5490192 +0000 UTC m=+4.138180611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs") pod "network-metrics-daemon-z4mc4" (UID: "6fe0d1a0-3adb-4b33-89b6-1f39fb282399") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:55:48.549442 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.549424 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:55:48.549513 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.549478 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret podName:e718a01a-548e-4499-95dc-7cbbd6ad6a12 nodeName:}" failed. No retries permitted until 2026-10-30 17:55:49.549462178 +0000 UTC m=+4.138623568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret") pod "global-pull-secret-syncer-2pp59" (UID: "e718a01a-548e-4499-95dc-7cbbd6ad6a12") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:55:48.650057 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.650020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjhv\" (UniqueName: \"kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv\") pod \"network-check-target-lzl5g\" (UID: \"92ba2348-2452-4bd2-92ba-a980fdc8a15f\") " pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:48.650204 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.650186 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:55:48.650249 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.650208 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:55:48.650249 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.650220 2573 projected.go:196] Error preparing data for projected volume kube-api-access-zxjhv for pod openshift-network-diagnostics/network-check-target-lzl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:55:48.650346 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:48.650277 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv podName:92ba2348-2452-4bd2-92ba-a980fdc8a15f nodeName:}" failed. No retries permitted until 2026-10-30 17:55:49.650258105 +0000 UTC m=+4.239419496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zxjhv" (UniqueName: "kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv") pod "network-check-target-lzl5g" (UID: "92ba2348-2452-4bd2-92ba-a980fdc8a15f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:55:48.898216 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.898149 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:50:48 +0000 UTC" deadline="2028-06-16 00:20:02.586655156 +0000 UTC" Oct 30 17:55:48.898216 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.898216 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14262h24m13.688443659s" Oct 30 17:55:48.912104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.912071 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" event={"ID":"e882582b-e97c-4171-8cff-eba17371f519","Type":"ContainerStarted","Data":"0c15a7e453d497f6a249ff157b6b3a54ba12fa006933cca34c9fe902edc35786"} Oct 30 17:55:48.913662 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.913611 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wv5gn" event={"ID":"167ba867-3860-40d7-9d55-9c81bf49930b","Type":"ContainerStarted","Data":"80ed2bfab374e0140e9c9253103658cb87017bfb519929d673b82c54d75efd4e"} Oct 30 17:55:48.924853 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.924822 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t92pz" event={"ID":"c3bd5eaa-c913-4709-8632-5abec1047a5e","Type":"ContainerStarted","Data":"b1f52a68261f62e38dd998161ceb6188a65e071c483afa4a3b43b1eb4f3ed105"} Oct 30 17:55:48.930848 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.930782 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bg76c" event={"ID":"6dc28a5a-54a6-4530-b383-4156404c6f4a","Type":"ContainerStarted","Data":"9d02664274cfb9bf9de77c00f6954e5b228ca3ba9ff4e19497d60a0c04a14a91"} Oct 30 17:55:48.935366 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.935342 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l67qw" event={"ID":"f4572363-4113-4e99-a34a-1b33a4cc08ff","Type":"ContainerStarted","Data":"6dc72b7434eeea4524819dde3b67eb31988432a1bd628c63952b50316e0946bc"} Oct 30 17:55:48.939272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.939192 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" event={"ID":"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d","Type":"ContainerStarted","Data":"33026e27915ca6ff982c8be6a7c493da5256310a700f13c21dd8208c396f5309"} Oct 30 17:55:48.943391 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.943364 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-94ttz" event={"ID":"f1043899-80a3-47a9-9d1c-0421cd68df00","Type":"ContainerStarted","Data":"9e2650d8ec1d7c969cf0fee983ebe7c48d0543299c11bcf27c6b7fe631b9ad15"} Oct 30 17:55:48.944972 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.944945 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" event={"ID":"13295b32-a6cc-4918-bfce-9e7819398cb0","Type":"ContainerStarted","Data":"12e41e5e05545f170de90c819a3692a0a7b2a6bf52372ecc4a7c0e7b7da34845"} Oct 30 17:55:48.951707 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.948564 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f5mwt" event={"ID":"47ef705c-0b43-46a9-82ba-fe9a8d426ff3","Type":"ContainerStarted","Data":"2e84fbad55f885a2bb894c95b6d4b4280cd7e6dadb02873fe5aa6d4d447e6201"} Oct 30 17:55:48.958594 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:48.958039 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" event={"ID":"cea7e9c368b2e2c6cdd7c0cacd3b509e","Type":"ContainerStarted","Data":"3106549653c623fc69a8e20c47d26294dfee858cda5bff51cab3b82b83e4cf1d"} Oct 30 17:55:49.559471 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:49.558737 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:49.559471 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:49.558824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:49.559471 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:49.558968 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:55:49.559471 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:49.559027 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs podName:6fe0d1a0-3adb-4b33-89b6-1f39fb282399 nodeName:}" failed. No retries permitted until 2026-10-30 17:55:51.55900662 +0000 UTC m=+6.148168007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs") pod "network-metrics-daemon-z4mc4" (UID: "6fe0d1a0-3adb-4b33-89b6-1f39fb282399") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:55:49.559471 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:49.559392 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:55:49.559471 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:49.559438 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret podName:e718a01a-548e-4499-95dc-7cbbd6ad6a12 nodeName:}" failed. No retries permitted until 2026-10-30 17:55:51.559423334 +0000 UTC m=+6.148584723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret") pod "global-pull-secret-syncer-2pp59" (UID: "e718a01a-548e-4499-95dc-7cbbd6ad6a12") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:55:49.661044 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:49.660966 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjhv\" (UniqueName: \"kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv\") pod \"network-check-target-lzl5g\" (UID: \"92ba2348-2452-4bd2-92ba-a980fdc8a15f\") " pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:49.661195 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:49.661120 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:55:49.661195 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:49.661144 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:55:49.661195 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:49.661156 2573 projected.go:196] Error preparing data for projected volume kube-api-access-zxjhv for pod openshift-network-diagnostics/network-check-target-lzl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:55:49.661360 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:49.661208 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv podName:92ba2348-2452-4bd2-92ba-a980fdc8a15f nodeName:}" failed. No retries permitted until 2026-10-30 17:55:51.661190786 +0000 UTC m=+6.250352185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zxjhv" (UniqueName: "kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv") pod "network-check-target-lzl5g" (UID: "92ba2348-2452-4bd2-92ba-a980fdc8a15f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:55:49.899596 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:49.899348 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:50:48 +0000 UTC" deadline="2028-07-02 09:43:05.376785455 +0000 UTC" Oct 30 17:55:49.899596 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:49.899610 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14655h47m15.477195322s" Oct 30 17:55:49.902184 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:49.902161 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:49.902306 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:49.902283 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:55:49.902858 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:49.902697 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:49.902858 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:49.902791 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:55:49.903255 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:49.903111 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:49.903255 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:49.903221 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:55:49.972733 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:49.972586 2573 generic.go:356] "Generic (PLEG): container finished" podID="f8da8176727de6d4893cff0cfaade9af" containerID="aedf5d7db9427ac1c9b084153689f10cd848a09f2eccd411285bc674c3231751" exitCode=0 Oct 30 17:55:49.973432 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:49.973407 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" event={"ID":"f8da8176727de6d4893cff0cfaade9af","Type":"ContainerDied","Data":"aedf5d7db9427ac1c9b084153689f10cd848a09f2eccd411285bc674c3231751"} Oct 30 17:55:50.018175 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:50.018122 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" podStartSLOduration=3.018105969 podStartE2EDuration="3.018105969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:49.004855455 +0000 UTC m=+3.594016862" watchObservedRunningTime="2026-10-30 17:55:50.018105969 +0000 UTC m=+4.607267375" Oct 30 17:55:51.001439 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:51.001407 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" event={"ID":"f8da8176727de6d4893cff0cfaade9af","Type":"ContainerStarted","Data":"9f7578f2df45ec7e47bcd62f57c351a4d99fbb380b3a5dd2065b4fab177450b5"} Oct 30 17:55:51.046801 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:51.046739 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" podStartSLOduration=4.046716797 podStartE2EDuration="4.046716797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:51.045941945 +0000 UTC m=+5.635103351" watchObservedRunningTime="2026-10-30 17:55:51.046716797 +0000 UTC m=+5.635878202" Oct 30 17:55:51.579918 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:51.579154 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:51.579918 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:51.579243 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:51.579918 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:51.579379 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:55:51.579918 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:51.579441 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs podName:6fe0d1a0-3adb-4b33-89b6-1f39fb282399 nodeName:}" failed. No retries permitted until 2026-10-30 17:55:55.579421237 +0000 UTC m=+10.168582635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs") pod "network-metrics-daemon-z4mc4" (UID: "6fe0d1a0-3adb-4b33-89b6-1f39fb282399") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:55:51.579918 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:51.579523 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:55:51.579918 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:51.579557 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret podName:e718a01a-548e-4499-95dc-7cbbd6ad6a12 nodeName:}" failed. No retries permitted until 2026-10-30 17:55:55.579545703 +0000 UTC m=+10.168707099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret") pod "global-pull-secret-syncer-2pp59" (UID: "e718a01a-548e-4499-95dc-7cbbd6ad6a12") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:55:51.681054 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:51.680434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjhv\" (UniqueName: \"kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv\") pod \"network-check-target-lzl5g\" (UID: \"92ba2348-2452-4bd2-92ba-a980fdc8a15f\") " pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:51.681054 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:51.680621 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:55:51.681054 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:51.680640 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:55:51.681054 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:51.680653 2573 projected.go:196] Error preparing data for projected volume kube-api-access-zxjhv for pod openshift-network-diagnostics/network-check-target-lzl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:55:51.681054 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:51.680707 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv podName:92ba2348-2452-4bd2-92ba-a980fdc8a15f nodeName:}" failed. No retries permitted until 2026-10-30 17:55:55.680689651 +0000 UTC m=+10.269851048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zxjhv" (UniqueName: "kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv") pod "network-check-target-lzl5g" (UID: "92ba2348-2452-4bd2-92ba-a980fdc8a15f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:55:51.903540 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:51.902704 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:51.903540 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:51.902729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:51.903540 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:51.902850 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:55:51.903540 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:51.902878 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:51.903540 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:51.902988 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:55:51.903540 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:51.903082 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:55:52.008954 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:52.008918 2573 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="e54e92ac0e79f62391f8e7ff4940f454e9f638b4f824e0ef2f830e2716024dd5" exitCode=0 Oct 30 17:55:52.009948 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:52.009912 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bg76c" event={"ID":"6dc28a5a-54a6-4530-b383-4156404c6f4a","Type":"ContainerDied","Data":"e54e92ac0e79f62391f8e7ff4940f454e9f638b4f824e0ef2f830e2716024dd5"} Oct 30 17:55:53.013363 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:53.013316 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wv5gn" event={"ID":"167ba867-3860-40d7-9d55-9c81bf49930b","Type":"ContainerStarted","Data":"57f5aae4b84ea520babf3f24fb82f8e6301256d748ecca604afc924571d3794b"} Oct 30 17:55:53.902613 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:53.902491 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:53.902765 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:53.902635 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:55:53.903077 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:53.903057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:53.903219 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:53.903182 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:55:53.903290 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:53.903276 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:53.903383 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:53.903365 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:55:54.018406 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:54.018362 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l67qw" event={"ID":"f4572363-4113-4e99-a34a-1b33a4cc08ff","Type":"ContainerStarted","Data":"2b60af2c96e81265319bbef04bac3e46d85621a536119761fc3c70b0d6852b69"} Oct 30 17:55:54.030367 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:54.030337 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" event={"ID":"13295b32-a6cc-4918-bfce-9e7819398cb0","Type":"ContainerStarted","Data":"3ad6c33110db2b43dc22919759d364257a538a26bfe48794d6be577b4938132a"} Oct 30 17:55:54.060991 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:54.060931 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wv5gn" podStartSLOduration=4.307978724 podStartE2EDuration="8.060910841s" totalImagesPullingTime="3.752932117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:53.078524263 +0000 UTC m=+7.667685666" watchObservedRunningTime="2026-10-30 17:55:54.060910841 +0000 UTC m=+8.650072246" Oct 30 17:55:55.079520 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.078790 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:55:55.080335 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.080141 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:55:55.110314 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.110182 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:55:55.123474 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.123424 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l67qw" podStartSLOduration=4.411212836 podStartE2EDuration="9.123408267s" totalImagesPullingTime="4.712195431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:54.061661788 +0000 UTC m=+8.650823223" watchObservedRunningTime="2026-10-30 17:55:55.123408267 +0000 UTC m=+9.712569674" Oct 30 17:55:55.618297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.617612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:55.618297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.617672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:55.618297 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:55.617797 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:55:55.618297 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:55.617853 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret podName:e718a01a-548e-4499-95dc-7cbbd6ad6a12 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:03.617837774 +0000 UTC m=+18.206999160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret") pod "global-pull-secret-syncer-2pp59" (UID: "e718a01a-548e-4499-95dc-7cbbd6ad6a12") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:55:55.618297 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:55.618232 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:55:55.618297 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:55.618266 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs podName:6fe0d1a0-3adb-4b33-89b6-1f39fb282399 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:03.61825584 +0000 UTC m=+18.207417223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs") pod "network-metrics-daemon-z4mc4" (UID: "6fe0d1a0-3adb-4b33-89b6-1f39fb282399") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:55:55.718845 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.718794 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjhv\" (UniqueName: \"kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv\") pod \"network-check-target-lzl5g\" (UID: \"92ba2348-2452-4bd2-92ba-a980fdc8a15f\") " pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:55.718995 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:55.718974 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:55:55.718995 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:55.718994 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:55:55.719125 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:55.719006 2573 projected.go:196] Error preparing data for projected volume kube-api-access-zxjhv for pod openshift-network-diagnostics/network-check-target-lzl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:55:55.719125 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:55.719054 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv podName:92ba2348-2452-4bd2-92ba-a980fdc8a15f nodeName:}" failed. No retries permitted until 2026-10-30 17:56:03.719036944 +0000 UTC m=+18.308198343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zxjhv" (UniqueName: "kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv") pod "network-check-target-lzl5g" (UID: "92ba2348-2452-4bd2-92ba-a980fdc8a15f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:55:55.901620 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.900634 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:55:55.110195914Z","UUID":"6096f397-2767-402b-a9d5-6cade2cfff04","Handler":null,"Name":"","Endpoint":""} Oct 30 17:55:55.904520 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.903672 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:55.904520 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:55.903778 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:55:55.904520 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.904094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:55.904520 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:55.904199 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:55:55.904520 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.904235 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:55.904520 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:55.904330 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:55:55.905710 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.905677 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:55:55.905710 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:55.905706 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:55:56.039326 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:56.039289 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" event={"ID":"13295b32-a6cc-4918-bfce-9e7819398cb0","Type":"ContainerStarted","Data":"d98dce5d5fd28eab1fe50ee8b653899023038b5cc3edd23627e58ca1f9ec180d"} Oct 30 17:55:56.042613 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:56.042535 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f5mwt" event={"ID":"47ef705c-0b43-46a9-82ba-fe9a8d426ff3","Type":"ContainerStarted","Data":"0bbff2c0a9070ffc613f64eede1cd2c59cc81383ee9dd952c73197986fa87ba8"} Oct 30 17:55:57.046962 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:57.046747 2573 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="8d4575654dc7cf557242654e8bb9726c6fa6c81adb46f339c5e13fe3c3fa34d1" exitCode=0 Oct 30 17:55:57.046962 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:57.046848 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bg76c" event={"ID":"6dc28a5a-54a6-4530-b383-4156404c6f4a","Type":"ContainerDied","Data":"8d4575654dc7cf557242654e8bb9726c6fa6c81adb46f339c5e13fe3c3fa34d1"} Oct 30 17:55:57.050009 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:57.049868 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" event={"ID":"13295b32-a6cc-4918-bfce-9e7819398cb0","Type":"ContainerStarted","Data":"59e91580d921381769bc81483f8a1781bfbe79d48f9163ac61e2e1bcf1472afd"} Oct 30 17:55:57.125890 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:57.124893 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-f5mwt" podStartSLOduration=3.640713935 podStartE2EDuration="11.124874667s" totalImagesPullingTime="7.484160732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:56.105570137 +0000 UTC m=+10.694731549" watchObservedRunningTime="2026-10-30 17:55:57.124874667 +0000 UTC m=+11.714036066" Oct 30 17:55:57.900892 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:57.900719 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:57.901062 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:57.900742 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:57.901062 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:57.900974 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:55:57.901174 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:57.900811 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:57.901174 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:57.901120 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:55:57.901268 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:57.901196 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:55:58.053554 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:58.053517 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t92pz" event={"ID":"c3bd5eaa-c913-4709-8632-5abec1047a5e","Type":"ContainerStarted","Data":"d9d216c93619d43560f7de3d15f59dee840c07ea4108dda088fa4b7b816ef438"} Oct 30 17:55:58.055401 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:58.055348 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-94ttz" event={"ID":"f1043899-80a3-47a9-9d1c-0421cd68df00","Type":"ContainerStarted","Data":"53419fca5143f43d9929b545796c6dfd24525dcd18cd09b5955d0ed179e72929"} Oct 30 17:55:58.096057 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:58.096016 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" podStartSLOduration=4.321840101 podStartE2EDuration="12.096004505s" totalImagesPullingTime="7.774164404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:55:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:57.171162171 +0000 UTC m=+11.760323575" watchObservedRunningTime="2026-10-30 17:55:58.096004505 +0000 UTC m=+12.685165909" Oct 30 17:55:58.096490 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:58.096459 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t92pz" podStartSLOduration=4.455847318 podStartE2EDuration="12.096453261s" totalImagesPullingTime="7.640605943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:58.095823563 +0000 UTC m=+12.684984971" watchObservedRunningTime="2026-10-30 17:55:58.096453261 +0000 UTC m=+12.685614667" Oct 30 17:55:59.060499 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:59.060465 2573 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="4c934ac67c6ce6bb75ca1a0469005bf547f3ba82f6839ad9102d65181f8a5276" exitCode=0 Oct 30 17:55:59.061106 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:59.060553 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bg76c" event={"ID":"6dc28a5a-54a6-4530-b383-4156404c6f4a","Type":"ContainerDied","Data":"4c934ac67c6ce6bb75ca1a0469005bf547f3ba82f6839ad9102d65181f8a5276"} Oct 30 17:55:59.062455 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:59.062395 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" event={"ID":"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d","Type":"ContainerStarted","Data":"db61992ae56b15edc782467d5455f8da3241a1744dace9b240c454541ff20682"} Oct 30 17:55:59.137125 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:59.137080 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-94ttz" podStartSLOduration=4.084452745 podStartE2EDuration="13.137067407s" totalImagesPullingTime="9.052614662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:58.137052729 +0000 UTC m=+12.726214136" watchObservedRunningTime="2026-10-30 17:55:59.137067407 +0000 UTC m=+13.726228833" Oct 30 17:55:59.181222 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:59.181180 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" podStartSLOduration=3.068717296 podStartE2EDuration="13.181165594s" totalImagesPullingTime="10.112448298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:59.180996824 +0000 UTC m=+13.770158245" watchObservedRunningTime="2026-10-30 17:55:59.181165594 +0000 UTC m=+13.770326998" Oct 30 17:55:59.901017 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:59.900986 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:55:59.901112 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:59.900994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:55:59.901176 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:59.901145 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:55:59.901288 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:55:59.901272 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:55:59.901401 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:59.901379 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:55:59.901483 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:55:59.901468 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:56:00.067218 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:00.067187 2573 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="0db10c05b12de4d42967e72bb78b5c297eb5d2e44f76950f71b5086079735ac7" exitCode=0 Oct 30 17:56:00.067890 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:00.067272 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bg76c" event={"ID":"6dc28a5a-54a6-4530-b383-4156404c6f4a","Type":"ContainerDied","Data":"0db10c05b12de4d42967e72bb78b5c297eb5d2e44f76950f71b5086079735ac7"} Oct 30 17:56:01.900671 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:01.900631 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:01.900671 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:01.900647 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:01.900671 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:01.900644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:01.901278 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:01.900771 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:56:01.901278 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:01.900805 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:56:01.901278 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:01.900880 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:56:03.076702 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.076533 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" event={"ID":"e882582b-e97c-4171-8cff-eba17371f519","Type":"ContainerStarted","Data":"7eba6212336199885b3495836cdcc164506429bdf28f11846d40dab44d51b249"} Oct 30 17:56:03.076991 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.076799 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" event={"ID":"e882582b-e97c-4171-8cff-eba17371f519","Type":"ContainerStarted","Data":"3d60efd6ba53b373a4378f3b216a58672fd3d242c7da90681d2536faf07c3a1d"} Oct 30 17:56:03.076991 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.076848 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" event={"ID":"e882582b-e97c-4171-8cff-eba17371f519","Type":"ContainerStarted","Data":"7c9e3bc1f6202eb5ceea04ba3855685fc56c8b5ed9e5ec704b4d4d18b70cd2ac"} Oct 30 17:56:03.076991 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.076878 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" event={"ID":"e882582b-e97c-4171-8cff-eba17371f519","Type":"ContainerStarted","Data":"1306183d1b5831029f5f13bec60ce15a6235c1a226cdb54b2aef4b693c4d2197"} Oct 30 17:56:03.076991 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.076922 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" event={"ID":"e882582b-e97c-4171-8cff-eba17371f519","Type":"ContainerStarted","Data":"84beb73a037e57f9e3befe12492c61894a2ea3056b57aa50d1bfe9023e07faf8"} Oct 30 17:56:03.079259 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.079236 2573 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="45a1ef7203ada532e26fb6e8c7625b591d99715f6faecdecdf7afadfb7d4e6d2" exitCode=0 Oct 30 17:56:03.079363 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.079276 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bg76c" event={"ID":"6dc28a5a-54a6-4530-b383-4156404c6f4a","Type":"ContainerDied","Data":"45a1ef7203ada532e26fb6e8c7625b591d99715f6faecdecdf7afadfb7d4e6d2"} Oct 30 17:56:03.683031 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.682819 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:03.683213 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.683063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:03.683213 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:03.682954 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:03.683213 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:03.683151 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret podName:e718a01a-548e-4499-95dc-7cbbd6ad6a12 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:19.683134463 +0000 UTC m=+34.272295846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret") pod "global-pull-secret-syncer-2pp59" (UID: "e718a01a-548e-4499-95dc-7cbbd6ad6a12") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:03.683213 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:03.683163 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:03.683213 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:03.683215 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs podName:6fe0d1a0-3adb-4b33-89b6-1f39fb282399 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:19.683197248 +0000 UTC m=+34.272358642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs") pod "network-metrics-daemon-z4mc4" (UID: "6fe0d1a0-3adb-4b33-89b6-1f39fb282399") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:03.783493 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.783472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjhv\" (UniqueName: \"kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv\") pod \"network-check-target-lzl5g\" (UID: \"92ba2348-2452-4bd2-92ba-a980fdc8a15f\") " pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:03.783631 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:03.783570 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:56:03.783631 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:03.783584 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:56:03.783631 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:03.783623 2573 projected.go:196] Error preparing data for projected volume kube-api-access-zxjhv for pod openshift-network-diagnostics/network-check-target-lzl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:03.783722 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:03.783663 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv podName:92ba2348-2452-4bd2-92ba-a980fdc8a15f nodeName:}" failed. No retries permitted until 2026-10-30 17:56:19.783653625 +0000 UTC m=+34.372815011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zxjhv" (UniqueName: "kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv") pod "network-check-target-lzl5g" (UID: "92ba2348-2452-4bd2-92ba-a980fdc8a15f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:03.900846 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.900817 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:03.900846 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.900836 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:03.900992 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:03.900949 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:56:03.900992 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:03.900975 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:56:03.901088 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.901002 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:03.901088 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:03.901064 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:56:03.903682 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.903659 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:56:03.903777 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:03.903734 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:56:04.085233 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:04.085207 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" event={"ID":"e882582b-e97c-4171-8cff-eba17371f519","Type":"ContainerStarted","Data":"84169b3d520f25bb6230f6b0219bcad2b48dc38fa0688c882aa3f9dbdbaf7f59"} Oct 30 17:56:04.088383 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:04.088357 2573 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="0be5b1a5e89414aa9942f542f33fed838e0015ecc385d605edb82b3d56993c5e" exitCode=0 Oct 30 17:56:04.088532 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:04.088443 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bg76c" event={"ID":"6dc28a5a-54a6-4530-b383-4156404c6f4a","Type":"ContainerDied","Data":"0be5b1a5e89414aa9942f542f33fed838e0015ecc385d605edb82b3d56993c5e"} Oct 30 17:56:05.093954 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:05.093922 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bg76c" event={"ID":"6dc28a5a-54a6-4530-b383-4156404c6f4a","Type":"ContainerStarted","Data":"78ff9f5b92684f69a578db4c3f8f6d3470575073460c193e07662ca60787087e"} Oct 30 17:56:05.170849 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:05.170796 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bg76c" podStartSLOduration=8.270906861 podStartE2EDuration="19.170780644s" totalImagesPullingTime="10.899873783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:55:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:05.170254269 +0000 UTC m=+19.759415673" watchObservedRunningTime="2026-10-30 17:56:05.170780644 +0000 UTC m=+19.759942048" Oct 30 17:56:05.901009 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:05.900978 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:05.901366 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:05.901326 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:56:05.901366 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:05.901102 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:05.901366 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:05.901116 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:05.901625 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:05.901445 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:56:05.901625 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:05.901512 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:56:06.099897 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:06.099864 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" event={"ID":"e882582b-e97c-4171-8cff-eba17371f519","Type":"ContainerStarted","Data":"438bc95036401f356a81e2f66668e128e254d9526644c1d292d0db6aa891f23c"} Oct 30 17:56:07.900447 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:07.900403 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:07.900447 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:07.900438 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:07.901176 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:07.900518 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:07.901176 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:07.900524 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:56:07.901176 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:07.900581 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:56:07.901176 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:07.900703 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:56:08.107239 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:08.107215 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" event={"ID":"e882582b-e97c-4171-8cff-eba17371f519","Type":"ContainerStarted","Data":"cafcbb15f41339a0fdcade9580c170dc02740e77e1a3bb843f16385379a8b2fc"} Oct 30 17:56:08.107521 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:08.107501 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:08.107632 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:08.107574 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:08.122919 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:08.122902 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:08.203925 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:08.203832 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" podStartSLOduration=7.678950403 podStartE2EDuration="22.203822032s" totalImagesPullingTime="14.524871629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:08.203361866 +0000 UTC m=+22.792523283" watchObservedRunningTime="2026-10-30 17:56:08.203822032 +0000 UTC m=+22.792983436" Oct 30 17:56:09.109213 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:09.109183 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:09.123907 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:09.123886 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:09.392991 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:09.392928 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2pp59"] Oct 30 17:56:09.393111 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:09.393016 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:09.393111 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:09.393099 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:56:09.393465 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:09.393435 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lzl5g"] Oct 30 17:56:09.393571 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:09.393524 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:09.393652 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:09.393633 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:56:09.395453 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:09.395430 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z4mc4"] Oct 30 17:56:09.395541 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:09.395527 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:09.395672 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:09.395654 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:56:10.900417 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:10.900385 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:10.900911 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:10.900453 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:10.900911 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:10.900542 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:10.900911 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:10.900547 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:56:10.900911 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:10.900626 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:56:10.900911 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:10.900687 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:56:12.900461 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:12.900279 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:12.900937 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:12.900338 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:12.900937 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:12.900536 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:56:12.900937 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:12.900338 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:12.900937 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:12.900628 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:56:12.900937 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:12.900682 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:56:14.900090 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:14.900046 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:14.900090 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:14.900080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:14.900579 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:14.900160 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:14.900579 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:14.900172 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzl5g" podUID="92ba2348-2452-4bd2-92ba-a980fdc8a15f" Oct 30 17:56:14.900579 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:14.900219 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pp59" podUID="e718a01a-548e-4499-95dc-7cbbd6ad6a12" Oct 30 17:56:14.900579 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:14.900335 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4mc4" podUID="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" Oct 30 17:56:15.225633 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.225562 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeReady" Oct 30 17:56:15.225727 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.225672 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 17:56:15.277413 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.277392 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dh588"] Oct 30 17:56:15.280483 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.280459 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dh588" Oct 30 17:56:15.281722 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.281698 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-htdwd"] Oct 30 17:56:15.284457 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.284438 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-j4bgx"] Oct 30 17:56:15.284583 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.284561 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:15.287163 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.287145 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9hqsk"] Oct 30 17:56:15.287286 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.287271 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.289975 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.289958 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4znfd"] Oct 30 17:56:15.290108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.290091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9hqsk" Oct 30 17:56:15.292626 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.292580 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c5cf884d7-2k5bz"] Oct 30 17:56:15.292732 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.292675 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4znfd" Oct 30 17:56:15.295588 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.295570 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4"] Oct 30 17:56:15.296316 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.296187 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.298454 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.298433 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn"] Oct 30 17:56:15.298570 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.298549 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:15.302075 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.302053 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5mn6l"] Oct 30 17:56:15.302164 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.302154 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn" Oct 30 17:56:15.304922 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.304904 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx"] Oct 30 17:56:15.305038 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.305019 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.307762 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.307741 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98"] Oct 30 17:56:15.307882 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.307862 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:15.310777 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.310760 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj"] Oct 30 17:56:15.310902 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.310894 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:15.314189 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.314139 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b849f747d-p4c4m"] Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.316917 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.317948 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.318030 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.318178 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.318326 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.318349 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.318656 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.318699 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bw9tq\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.318771 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.318956 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.319003 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.319076 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.319377 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.319467 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-m9gkv\"" Oct 30 17:56:15.321501 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.319794 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:15.324854 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.324830 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-mjzvg\"" Oct 30 17:56:15.328683 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.328664 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-698f87866c-k4grf"] Oct 30 17:56:15.328866 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.328847 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.330745 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.330723 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 17:56:15.331112 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.331091 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 17:56:15.331675 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.331658 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c"] Oct 30 17:56:15.333657 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.333639 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.336561 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.336456 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-867jv"] Oct 30 17:56:15.336847 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.336814 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c" Oct 30 17:56:15.340380 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.340351 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vj5k4"] Oct 30 17:56:15.340508 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.340466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.343354 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.343332 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dh588"] Oct 30 17:56:15.343461 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.343391 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-htdwd"] Oct 30 17:56:15.343461 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.343414 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9hqsk"] Oct 30 17:56:15.343461 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.343422 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:15.343461 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.343445 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4"] Oct 30 17:56:15.343461 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.343464 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-j4bgx"] Oct 30 17:56:15.343750 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.343484 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx"] Oct 30 17:56:15.343750 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.343503 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4znfd"] Oct 30 17:56:15.343750 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.343520 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5mn6l"] Oct 30 17:56:15.343750 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.343537 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c"] Oct 30 17:56:15.343750 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.343551 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj"] Oct 30 17:56:15.345349 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.345325 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vj5k4"] Oct 30 17:56:15.346099 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.346083 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b849f747d-p4c4m"] Oct 30 17:56:15.351988 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.351966 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn"] Oct 30 17:56:15.357623 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.357196 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98"] Oct 30 17:56:15.357623 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.357234 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c5cf884d7-2k5bz"] Oct 30 17:56:15.359674 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.359655 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-867jv"] Oct 30 17:56:15.363021 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.362998 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-698f87866c-k4grf"] Oct 30 17:56:15.371123 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.371214 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjng\" (UniqueName: \"kubernetes.io/projected/2b407f39-23cc-432f-a5d7-3a377e4762a5-kube-api-access-jnjng\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.371272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qld6b\" (UniqueName: \"kubernetes.io/projected/a97de0f2-2248-44e3-bee7-9817777a261d-kube-api-access-qld6b\") pod \"kas-connection-checker-685895c7c9-9hqsk\" (UID: \"a97de0f2-2248-44e3-bee7-9817777a261d\") " pod="kube-system/kas-connection-checker-685895c7c9-9hqsk" Oct 30 17:56:15.371272 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-snapshots\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.371382 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blc54\" (UniqueName: \"kubernetes.io/projected/38a4b7b8-168f-4525-81be-d43ef28e4c13-kube-api-access-blc54\") pod \"kas-connection-checker-685895c7c9-dh588\" (UID: \"38a4b7b8-168f-4525-81be-d43ef28e4c13\") " pod="kube-system/kas-connection-checker-685895c7c9-dh588" Oct 30 17:56:15.371427 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371390 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.371427 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-service-ca-bundle\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.371427 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4cdl\" (UniqueName: \"kubernetes.io/projected/f2070749-ec46-433b-975e-470c734e5fe5-kube-api-access-h4cdl\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.371572 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371441 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrpz\" (UniqueName: \"kubernetes.io/projected/bb2c63b2-119c-4f2e-a7e5-664bf9e9b999-kube-api-access-hnrpz\") pod \"volume-data-source-validator-58c9f5f46d-ggqkn\" (UID: \"bb2c63b2-119c-4f2e-a7e5-664bf9e9b999\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn" Oct 30 17:56:15.371572 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-default-certificate\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.371572 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371500 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-stats-auth\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.371572 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371529 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/de9d2339-d6be-4332-921d-ec5717b06699-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:15.371572 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:15.371841 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371577 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmms\" (UniqueName: \"kubernetes.io/projected/e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc-kube-api-access-jdmms\") pod \"kas-connection-checker-685895c7c9-4znfd\" (UID: \"e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc\") " pod="kube-system/kas-connection-checker-685895c7c9-4znfd" Oct 30 17:56:15.371841 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cs2d\" (UniqueName: \"kubernetes.io/projected/de9d2339-d6be-4332-921d-ec5717b06699-kube-api-access-8cs2d\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:15.371841 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371679 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f53a361f-64db-41a3-abd1-c04251831048-nginx-conf\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:15.371841 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-tmp\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.371841 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2070749-ec46-433b-975e-470c734e5fe5-serving-cert\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.371841 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.371841 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.371769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:15.413969 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.413951 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:56:15.414067 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.414024 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 17:56:15.414067 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.414024 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:56:15.414067 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.414046 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 17:56:15.414226 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.414164 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 17:56:15.414226 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.414173 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-f7vbx\"" Oct 30 17:56:15.414226 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.414185 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 17:56:15.414226 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.414199 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:15.414226 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.414191 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 17:56:15.414226 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.414217 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:15.414226 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.414220 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:56:15.415339 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415325 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 17:56:15.415544 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415533 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:56:15.415623 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415566 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:56:15.415685 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415654 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 17:56:15.415685 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415658 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:56:15.415685 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415680 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 17:56:15.415800 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415680 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4qvdt\"" Oct 30 17:56:15.415800 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415704 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 17:56:15.415800 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415707 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dwcjh\"" Oct 30 17:56:15.415800 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415739 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:15.415913 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415884 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 17:56:15.415948 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415925 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:56:15.415948 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415935 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bn4bk\"" Oct 30 17:56:15.416006 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415952 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 17:56:15.416006 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415934 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:15.416006 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415978 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:56:15.416006 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415986 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7j6rv\"" Oct 30 17:56:15.416006 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415978 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksmnn\"" Oct 30 17:56:15.416006 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.415997 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:56:15.416174 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416060 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 17:56:15.416174 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416080 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:15.416247 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416234 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:15.416247 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416242 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:15.416308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416290 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:15.416308 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416275 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:15.416387 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416312 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:15.416387 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416379 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tswqq\"" Oct 30 17:56:15.416475 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416379 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wj68l\"" Oct 30 17:56:15.416475 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416409 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-txfz9\"" Oct 30 17:56:15.416565 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416508 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 17:56:15.416565 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416548 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-p2k4l\"" Oct 30 17:56:15.416979 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.416967 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 17:56:15.430440 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.430417 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 17:56:15.430566 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.430549 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 17:56:15.472061 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472041 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-config\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.472132 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-bound-sa-token\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.472132 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b26b3373-a3b2-4a0d-bc89-94bd729b849e-config-volume\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.472132 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26b3373-a3b2-4a0d-bc89-94bd729b849e-tmp-dir\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.472132 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.472289 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472173 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/551adda7-a285-4f6e-9daf-b5b9f4410b48-config\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:15.472289 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472194 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-installation-pull-secrets\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.472289 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472268 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c06084-1bed-46c5-a576-f6129e21e062-ca-trust-extracted\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.472505 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.472319 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:15.972303109 +0000 UTC m=+30.561464511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : configmap references non-existent config key: service-ca.crt Oct 30 17:56:15.472505 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjng\" (UniqueName: \"kubernetes.io/projected/2b407f39-23cc-432f-a5d7-3a377e4762a5-kube-api-access-jnjng\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.472505 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472437 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskfk\" (UniqueName: \"kubernetes.io/projected/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-kube-api-access-bskfk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:15.472505 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-trusted-ca\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.472505 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5c28\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-kube-api-access-t5c28\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.472783 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9pgj\" (UniqueName: \"kubernetes.io/projected/551adda7-a285-4f6e-9daf-b5b9f4410b48-kube-api-access-z9pgj\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:15.472783 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-default-certificate\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.472783 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472682 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-stats-auth\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.472783 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472710 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/de9d2339-d6be-4332-921d-ec5717b06699-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:15.472783 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.472783 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472751 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/551adda7-a285-4f6e-9daf-b5b9f4410b48-serving-cert\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:15.472783 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472767 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-bound-sa-token\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.473108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blc54\" (UniqueName: \"kubernetes.io/projected/38a4b7b8-168f-4525-81be-d43ef28e4c13-kube-api-access-blc54\") pod \"kas-connection-checker-685895c7c9-dh588\" (UID: \"38a4b7b8-168f-4525-81be-d43ef28e4c13\") " pod="kube-system/kas-connection-checker-685895c7c9-dh588" Oct 30 17:56:15.473108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-service-ca-bundle\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.473108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472925 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:15.473108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-trusted-ca\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.473108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.472978 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k9gj\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-kube-api-access-4k9gj\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.473108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:15.473108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmms\" (UniqueName: \"kubernetes.io/projected/e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc-kube-api-access-jdmms\") pod \"kas-connection-checker-685895c7c9-4znfd\" (UID: \"e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc\") " pod="kube-system/kas-connection-checker-685895c7c9-4znfd" Oct 30 17:56:15.473108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-trusted-ca\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.473464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29549d27-df3a-48d0-8e7b-85fb883b4481-serving-cert\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.473464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473142 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-image-registry-private-configuration\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.473464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:15.473464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473225 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcrv\" (UniqueName: \"kubernetes.io/projected/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-kube-api-access-qfcrv\") pod \"ingress-canary-vj5k4\" (UID: \"de504e55-5beb-4f9a-a08a-b4f4e1f999d6\") " pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:15.473464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.473464 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.473419 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:15.473725 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2070749-ec46-433b-975e-470c734e5fe5-serving-cert\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.473725 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.473489 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls podName:de9d2339-d6be-4332-921d-ec5717b06699 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:15.973471194 +0000 UTC m=+30.562632593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wb2x4" (UID: "de9d2339-d6be-4332-921d-ec5717b06699") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:15.473725 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1136549d-7491-452c-b396-f839668b5a19-ca-trust-extracted\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.473725 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.473584 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:56:15.473725 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473589 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-service-ca-bundle\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.473725 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473624 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/de9d2339-d6be-4332-921d-ec5717b06699-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:15.473725 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.473638 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:15.973626669 +0000 UTC m=+30.562788065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : secret "router-metrics-certs-default" not found Oct 30 17:56:15.473725 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473633 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.473725 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hrbq\" (UniqueName: \"kubernetes.io/projected/e7e6e71c-794e-40ba-acef-7fddf496d546-kube-api-access-9hrbq\") pod \"network-check-source-6f78d59f99-5ks6c\" (UID: \"e7e6e71c-794e-40ba-acef-7fddf496d546\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c" Oct 30 17:56:15.474109 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473745 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-cert\") pod \"ingress-canary-vj5k4\" (UID: \"de504e55-5beb-4f9a-a08a-b4f4e1f999d6\") " pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:15.474109 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473779 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-installation-pull-secrets\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.474109 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473888 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qld6b\" (UniqueName: \"kubernetes.io/projected/a97de0f2-2248-44e3-bee7-9817777a261d-kube-api-access-qld6b\") pod \"kas-connection-checker-685895c7c9-9hqsk\" (UID: \"a97de0f2-2248-44e3-bee7-9817777a261d\") " pod="kube-system/kas-connection-checker-685895c7c9-9hqsk" Oct 30 17:56:15.474109 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.473942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-registry-certificates\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.474301 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbxf\" (UniqueName: \"kubernetes.io/projected/b26b3373-a3b2-4a0d-bc89-94bd729b849e-kube-api-access-wpbxf\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.474301 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474246 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrpz\" (UniqueName: \"kubernetes.io/projected/bb2c63b2-119c-4f2e-a7e5-664bf9e9b999-kube-api-access-hnrpz\") pod \"volume-data-source-validator-58c9f5f46d-ggqkn\" (UID: \"bb2c63b2-119c-4f2e-a7e5-664bf9e9b999\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn" Oct 30 17:56:15.474301 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f53a361f-64db-41a3-abd1-c04251831048-nginx-conf\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:15.474301 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:15.474500 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-snapshots\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.474500 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cdl\" (UniqueName: \"kubernetes.io/projected/f2070749-ec46-433b-975e-470c734e5fe5-kube-api-access-h4cdl\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.475026 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-image-registry-private-configuration\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.475026 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2h7h\" (UniqueName: \"kubernetes.io/projected/29549d27-df3a-48d0-8e7b-85fb883b4481-kube-api-access-d2h7h\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.475026 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474825 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnfj\" (UniqueName: \"kubernetes.io/projected/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-kube-api-access-ghnfj\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:15.475026 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474860 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:15.475026 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.474876 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:56:15.475026 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.475026 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:15.475026 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.474936 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert podName:f53a361f-64db-41a3-abd1-c04251831048 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:15.974920732 +0000 UTC m=+30.564082127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htdwd" (UID: "f53a361f-64db-41a3-abd1-c04251831048") : secret "networking-console-plugin-cert" not found Oct 30 17:56:15.475026 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.474986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cs2d\" (UniqueName: \"kubernetes.io/projected/de9d2339-d6be-4332-921d-ec5717b06699-kube-api-access-8cs2d\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:15.475526 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.475031 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.475526 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.475056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.475526 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.475085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-registry-certificates\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.475526 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.475122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-tmp\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.475526 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.475262 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f53a361f-64db-41a3-abd1-c04251831048-nginx-conf\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:15.475526 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.475484 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-snapshots\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.475916 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.475559 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-tmp\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.477019 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.476998 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2070749-ec46-433b-975e-470c734e5fe5-serving-cert\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.477428 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.477408 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-stats-auth\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.477496 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.477413 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-default-certificate\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.499842 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.499798 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjng\" (UniqueName: \"kubernetes.io/projected/2b407f39-23cc-432f-a5d7-3a377e4762a5-kube-api-access-jnjng\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.500700 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.500677 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qld6b\" (UniqueName: \"kubernetes.io/projected/a97de0f2-2248-44e3-bee7-9817777a261d-kube-api-access-qld6b\") pod \"kas-connection-checker-685895c7c9-9hqsk\" (UID: \"a97de0f2-2248-44e3-bee7-9817777a261d\") " pod="kube-system/kas-connection-checker-685895c7c9-9hqsk" Oct 30 17:56:15.500905 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.500881 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmms\" (UniqueName: \"kubernetes.io/projected/e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc-kube-api-access-jdmms\") pod \"kas-connection-checker-685895c7c9-4znfd\" (UID: \"e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc\") " pod="kube-system/kas-connection-checker-685895c7c9-4znfd" Oct 30 17:56:15.500974 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.500906 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cs2d\" (UniqueName: \"kubernetes.io/projected/de9d2339-d6be-4332-921d-ec5717b06699-kube-api-access-8cs2d\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:15.501076 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.501055 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4cdl\" (UniqueName: \"kubernetes.io/projected/f2070749-ec46-433b-975e-470c734e5fe5-kube-api-access-h4cdl\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.501130 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.501115 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blc54\" (UniqueName: \"kubernetes.io/projected/38a4b7b8-168f-4525-81be-d43ef28e4c13-kube-api-access-blc54\") pod \"kas-connection-checker-685895c7c9-dh588\" (UID: \"38a4b7b8-168f-4525-81be-d43ef28e4c13\") " pod="kube-system/kas-connection-checker-685895c7c9-dh588" Oct 30 17:56:15.501280 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.501267 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrpz\" (UniqueName: \"kubernetes.io/projected/bb2c63b2-119c-4f2e-a7e5-664bf9e9b999-kube-api-access-hnrpz\") pod \"volume-data-source-validator-58c9f5f46d-ggqkn\" (UID: \"bb2c63b2-119c-4f2e-a7e5-664bf9e9b999\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn" Oct 30 17:56:15.575675 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.575650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.575675 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.575675 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.575795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.575693 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-registry-certificates\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.575795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.575709 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-config\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.575795 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.575726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-bound-sa-token\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.575910 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.575793 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:56:15.575910 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.575812 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b849f747d-p4c4m: secret "image-registry-tls" not found Oct 30 17:56:15.575910 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.575843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b26b3373-a3b2-4a0d-bc89-94bd729b849e-config-volume\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.575910 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.575859 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls podName:1136549d-7491-452c-b396-f839668b5a19 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:16.075842576 +0000 UTC m=+30.665003973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls") pod "image-registry-5b849f747d-p4c4m" (UID: "1136549d-7491-452c-b396-f839668b5a19") : secret "image-registry-tls" not found Oct 30 17:56:15.575910 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.575892 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:56:15.575910 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.575909 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698f87866c-k4grf: secret "image-registry-tls" not found Oct 30 17:56:15.576190 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.575944 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls podName:78c06084-1bed-46c5-a576-f6129e21e062 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:16.075933987 +0000 UTC m=+30.665095382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls") pod "image-registry-698f87866c-k4grf" (UID: "78c06084-1bed-46c5-a576-f6129e21e062") : secret "image-registry-tls" not found Oct 30 17:56:15.576190 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.575890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26b3373-a3b2-4a0d-bc89-94bd729b849e-tmp-dir\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.576190 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.575983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/551adda7-a285-4f6e-9daf-b5b9f4410b48-config\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:15.576190 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-installation-pull-secrets\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.576190 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c06084-1bed-46c5-a576-f6129e21e062-ca-trust-extracted\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.576190 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskfk\" (UniqueName: \"kubernetes.io/projected/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-kube-api-access-bskfk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:15.576190 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-trusted-ca\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.576190 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5c28\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-kube-api-access-t5c28\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.576190 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9pgj\" (UniqueName: \"kubernetes.io/projected/551adda7-a285-4f6e-9daf-b5b9f4410b48-kube-api-access-z9pgj\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:15.576190 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576172 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/551adda7-a285-4f6e-9daf-b5b9f4410b48-serving-cert\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-bound-sa-token\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576237 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26b3373-a3b2-4a0d-bc89-94bd729b849e-tmp-dir\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-trusted-ca\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576314 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-registry-certificates\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576324 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k9gj\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-kube-api-access-4k9gj\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-trusted-ca\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29549d27-df3a-48d0-8e7b-85fb883b4481-serving-cert\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576442 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-config\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-image-registry-private-configuration\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576486 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576510 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcrv\" (UniqueName: \"kubernetes.io/projected/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-kube-api-access-qfcrv\") pod \"ingress-canary-vj5k4\" (UID: \"de504e55-5beb-4f9a-a08a-b4f4e1f999d6\") " pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576556 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b26b3373-a3b2-4a0d-bc89-94bd729b849e-config-volume\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576570 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/551adda7-a285-4f6e-9daf-b5b9f4410b48-config\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:15.576898 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.576745 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c06084-1bed-46c5-a576-f6129e21e062-ca-trust-extracted\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.577696 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.577506 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-trusted-ca\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.577696 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.577539 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-trusted-ca\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.577696 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.577618 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1136549d-7491-452c-b396-f839668b5a19-ca-trust-extracted\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.577696 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.577629 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-trusted-ca\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.577696 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.577661 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hrbq\" (UniqueName: \"kubernetes.io/projected/e7e6e71c-794e-40ba-acef-7fddf496d546-kube-api-access-9hrbq\") pod \"network-check-source-6f78d59f99-5ks6c\" (UID: \"e7e6e71c-794e-40ba-acef-7fddf496d546\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c" Oct 30 17:56:15.578136 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.578008 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:56:15.578233 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-cert\") pod \"ingress-canary-vj5k4\" (UID: \"de504e55-5beb-4f9a-a08a-b4f4e1f999d6\") " pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:15.578233 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-installation-pull-secrets\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.578340 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-registry-certificates\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.578340 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578322 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1136549d-7491-452c-b396-f839668b5a19-ca-trust-extracted\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbxf\" (UniqueName: \"kubernetes.io/projected/b26b3373-a3b2-4a0d-bc89-94bd729b849e-kube-api-access-wpbxf\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-image-registry-private-configuration\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2h7h\" (UniqueName: \"kubernetes.io/projected/29549d27-df3a-48d0-8e7b-85fb883b4481-kube-api-access-d2h7h\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.578642 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls podName:27d2c28b-a2b2-4293-ad25-a0e69b939b9c nodeName:}" failed. No retries permitted until 2026-10-30 17:56:16.078623984 +0000 UTC m=+30.667785388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgr98" (UID: "27d2c28b-a2b2-4293-ad25-a0e69b939b9c") : secret "samples-operator-tls" not found Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnfj\" (UniqueName: \"kubernetes.io/projected/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-kube-api-access-ghnfj\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578846 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-installation-pull-secrets\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.578700 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-registry-certificates\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.579324 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.579370 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls podName:b26b3373-a3b2-4a0d-bc89-94bd729b849e nodeName:}" failed. No retries permitted until 2026-10-30 17:56:16.079355505 +0000 UTC m=+30.668516901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls") pod "dns-default-867jv" (UID: "b26b3373-a3b2-4a0d-bc89-94bd729b849e") : secret "dns-default-metrics-tls" not found Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.579368 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.579405 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-image-registry-private-configuration\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.579464 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.579447 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:15.580461 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.579552 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:15.580461 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.580203 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29549d27-df3a-48d0-8e7b-85fb883b4481-serving-cert\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.581097 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.581070 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/551adda7-a285-4f6e-9daf-b5b9f4410b48-serving-cert\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:15.581202 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.581182 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-cert\") pod \"ingress-canary-vj5k4\" (UID: \"de504e55-5beb-4f9a-a08a-b4f4e1f999d6\") " pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:15.581666 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.581643 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-installation-pull-secrets\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.581732 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.581702 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-image-registry-private-configuration\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.594553 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.594531 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dh588" Oct 30 17:56:15.600641 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.600618 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-bound-sa-token\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.601025 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.601006 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5c28\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-kube-api-access-t5c28\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:15.601169 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.601152 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9pgj\" (UniqueName: \"kubernetes.io/projected/551adda7-a285-4f6e-9daf-b5b9f4410b48-kube-api-access-z9pgj\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:15.601300 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.601286 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k9gj\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-kube-api-access-4k9gj\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.606988 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.606962 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hrbq\" (UniqueName: \"kubernetes.io/projected/e7e6e71c-794e-40ba-acef-7fddf496d546-kube-api-access-9hrbq\") pod \"network-check-source-6f78d59f99-5ks6c\" (UID: \"e7e6e71c-794e-40ba-acef-7fddf496d546\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c" Oct 30 17:56:15.607189 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.607169 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskfk\" (UniqueName: \"kubernetes.io/projected/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-kube-api-access-bskfk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:15.607399 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.607382 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-bound-sa-token\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:15.608382 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.608353 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcrv\" (UniqueName: \"kubernetes.io/projected/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-kube-api-access-qfcrv\") pod \"ingress-canary-vj5k4\" (UID: \"de504e55-5beb-4f9a-a08a-b4f4e1f999d6\") " pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:15.608904 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.608881 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:15.610033 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.610012 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnfj\" (UniqueName: \"kubernetes.io/projected/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-kube-api-access-ghnfj\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:15.611242 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.611223 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbxf\" (UniqueName: \"kubernetes.io/projected/b26b3373-a3b2-4a0d-bc89-94bd729b849e-kube-api-access-wpbxf\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:15.611322 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.611302 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2h7h\" (UniqueName: \"kubernetes.io/projected/29549d27-df3a-48d0-8e7b-85fb883b4481-kube-api-access-d2h7h\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.618191 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.618172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9hqsk" Oct 30 17:56:15.630948 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.630924 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4znfd" Oct 30 17:56:15.657489 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.657457 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn" Oct 30 17:56:15.666577 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.666360 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:15.673189 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.673031 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:15.685893 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.685863 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:15.704258 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.703904 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c" Oct 30 17:56:15.727850 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.718275 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:15.917481 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.917459 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dh588"] Oct 30 17:56:15.919135 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:15.919112 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a4b7b8_168f_4525_81be_d43ef28e4c13.slice/crio-ec7e1df591d152e0d8408d32e6f8beee904d14190af055ba81d5c90cbcd22d04 WatchSource:0}: Error finding container ec7e1df591d152e0d8408d32e6f8beee904d14190af055ba81d5c90cbcd22d04: Status 404 returned error can't find the container with id ec7e1df591d152e0d8408d32e6f8beee904d14190af055ba81d5c90cbcd22d04 Oct 30 17:56:15.950247 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.950217 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4znfd"] Oct 30 17:56:15.954876 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.953555 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9hqsk"] Oct 30 17:56:15.961021 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.960668 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-j4bgx"] Oct 30 17:56:15.986517 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.985351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.986517 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.985441 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:15.986517 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.985516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:15.986517 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:15.985583 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:15.986517 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.985779 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:56:15.986517 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.985829 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert podName:f53a361f-64db-41a3-abd1-c04251831048 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:16.985811889 +0000 UTC m=+31.574973290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htdwd" (UID: "f53a361f-64db-41a3-abd1-c04251831048") : secret "networking-console-plugin-cert" not found Oct 30 17:56:15.986517 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.985899 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:16.98588837 +0000 UTC m=+31.575049769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : configmap references non-existent config key: service-ca.crt Oct 30 17:56:15.986517 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.985952 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:15.986517 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.985983 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls podName:de9d2339-d6be-4332-921d-ec5717b06699 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:16.985973189 +0000 UTC m=+31.575134571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wb2x4" (UID: "de9d2339-d6be-4332-921d-ec5717b06699") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:15.986517 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.986040 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:56:15.986517 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:15.986067 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:16.986058035 +0000 UTC m=+31.575219434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : secret "router-metrics-certs-default" not found Oct 30 17:56:16.007006 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.006876 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn"] Oct 30 17:56:16.011251 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:16.011226 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2c63b2_119c_4f2e_a7e5_664bf9e9b999.slice/crio-d3f8aa4f6af4a3c34261eaa5558e9617b146e6b1f6cdf7a4e0b124072e917f5f WatchSource:0}: Error finding container d3f8aa4f6af4a3c34261eaa5558e9617b146e6b1f6cdf7a4e0b124072e917f5f: Status 404 returned error can't find the container with id d3f8aa4f6af4a3c34261eaa5558e9617b146e6b1f6cdf7a4e0b124072e917f5f Oct 30 17:56:16.034163 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.034105 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx"] Oct 30 17:56:16.037564 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:16.037535 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551adda7_a285_4f6e_9daf_b5b9f4410b48.slice/crio-6eeef5b820286f6ffc3e108c52811f157c97b0effc4a4484ad24bcf1a0d8a2fd WatchSource:0}: Error finding container 6eeef5b820286f6ffc3e108c52811f157c97b0effc4a4484ad24bcf1a0d8a2fd: Status 404 returned error can't find the container with id 6eeef5b820286f6ffc3e108c52811f157c97b0effc4a4484ad24bcf1a0d8a2fd Oct 30 17:56:16.048522 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.048466 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5mn6l"] Oct 30 17:56:16.060237 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:16.059464 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29549d27_df3a_48d0_8e7b_85fb883b4481.slice/crio-7617edbac47192048c6e4eaf4cdfbcf583584596e790eb4f45e2c0f6f8f3a363 WatchSource:0}: Error finding container 7617edbac47192048c6e4eaf4cdfbcf583584596e790eb4f45e2c0f6f8f3a363: Status 404 returned error can't find the container with id 7617edbac47192048c6e4eaf4cdfbcf583584596e790eb4f45e2c0f6f8f3a363 Oct 30 17:56:16.088253 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.086764 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:16.088253 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.087105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:16.088253 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.087172 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:16.088253 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.087210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:16.090818 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.089818 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:56:16.090818 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.090144 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj"] Oct 30 17:56:16.090818 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.090217 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls podName:27d2c28b-a2b2-4293-ad25-a0e69b939b9c nodeName:}" failed. No retries permitted until 2026-10-30 17:56:17.090190975 +0000 UTC m=+31.679352371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgr98" (UID: "27d2c28b-a2b2-4293-ad25-a0e69b939b9c") : secret "samples-operator-tls" not found Oct 30 17:56:16.090818 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.090711 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:56:16.090818 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.090780 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:56:16.090818 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.090793 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698f87866c-k4grf: secret "image-registry-tls" not found Oct 30 17:56:16.091113 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.090860 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls podName:78c06084-1bed-46c5-a576-f6129e21e062 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:17.090839184 +0000 UTC m=+31.680000579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls") pod "image-registry-698f87866c-k4grf" (UID: "78c06084-1bed-46c5-a576-f6129e21e062") : secret "image-registry-tls" not found Oct 30 17:56:16.091253 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.091223 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls podName:b26b3373-a3b2-4a0d-bc89-94bd729b849e nodeName:}" failed. No retries permitted until 2026-10-30 17:56:17.091144021 +0000 UTC m=+31.680305417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls") pod "dns-default-867jv" (UID: "b26b3373-a3b2-4a0d-bc89-94bd729b849e") : secret "dns-default-metrics-tls" not found Oct 30 17:56:16.091323 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.090736 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:56:16.091323 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.091311 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b849f747d-p4c4m: secret "image-registry-tls" not found Oct 30 17:56:16.091418 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.091364 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls podName:1136549d-7491-452c-b396-f839668b5a19 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:17.091344161 +0000 UTC m=+31.680505560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls") pod "image-registry-5b849f747d-p4c4m" (UID: "1136549d-7491-452c-b396-f839668b5a19") : secret "image-registry-tls" not found Oct 30 17:56:16.096304 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.096274 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vj5k4"] Oct 30 17:56:16.097758 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:16.097709 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde504e55_5beb_4f9a_a08a_b4f4e1f999d6.slice/crio-bd720384799e29bce3b473e206fdb2afa4f4010ee72a5faf40f105472a4f29ea WatchSource:0}: Error finding container bd720384799e29bce3b473e206fdb2afa4f4010ee72a5faf40f105472a4f29ea: Status 404 returned error can't find the container with id bd720384799e29bce3b473e206fdb2afa4f4010ee72a5faf40f105472a4f29ea Oct 30 17:56:16.102520 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.102500 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c"] Oct 30 17:56:16.104210 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:16.104190 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e6e71c_794e_40ba_acef_7fddf496d546.slice/crio-c4cd751abe6f3d1bd40ebaeda185555b87ec00793cc51aea19aa28630130dd0b WatchSource:0}: Error finding container c4cd751abe6f3d1bd40ebaeda185555b87ec00793cc51aea19aa28630130dd0b: Status 404 returned error can't find the container with id c4cd751abe6f3d1bd40ebaeda185555b87ec00793cc51aea19aa28630130dd0b Oct 30 17:56:16.125279 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.125258 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dh588" event={"ID":"38a4b7b8-168f-4525-81be-d43ef28e4c13","Type":"ContainerStarted","Data":"6937ad6ac07d2799e26994e65518b4412e8e7d2a2fae26fb77effd4722ea01e1"} Oct 30 17:56:16.125382 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.125296 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dh588" event={"ID":"38a4b7b8-168f-4525-81be-d43ef28e4c13","Type":"ContainerStarted","Data":"ec7e1df591d152e0d8408d32e6f8beee904d14190af055ba81d5c90cbcd22d04"} Oct 30 17:56:16.126510 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.126485 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" event={"ID":"551adda7-a285-4f6e-9daf-b5b9f4410b48","Type":"ContainerStarted","Data":"6eeef5b820286f6ffc3e108c52811f157c97b0effc4a4484ad24bcf1a0d8a2fd"} Oct 30 17:56:16.127848 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.127825 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" event={"ID":"29549d27-df3a-48d0-8e7b-85fb883b4481","Type":"ContainerStarted","Data":"7617edbac47192048c6e4eaf4cdfbcf583584596e790eb4f45e2c0f6f8f3a363"} Oct 30 17:56:16.129108 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.129083 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vj5k4" event={"ID":"de504e55-5beb-4f9a-a08a-b4f4e1f999d6","Type":"ContainerStarted","Data":"bd720384799e29bce3b473e206fdb2afa4f4010ee72a5faf40f105472a4f29ea"} Oct 30 17:56:16.130291 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.130271 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn" event={"ID":"bb2c63b2-119c-4f2e-a7e5-664bf9e9b999","Type":"ContainerStarted","Data":"d3f8aa4f6af4a3c34261eaa5558e9617b146e6b1f6cdf7a4e0b124072e917f5f"} Oct 30 17:56:16.132088 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.132058 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4znfd" event={"ID":"e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc","Type":"ContainerStarted","Data":"c3b28c4fbd9eb69c7a6f0c16b6a0746927aed4b7034ee2df53631d1f5ae2eee6"} Oct 30 17:56:16.132167 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.132097 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4znfd" event={"ID":"e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc","Type":"ContainerStarted","Data":"e22cc8ce73b0d3b02e3bfcf34809f2fce345dae9b1760d39764bd95f4332970a"} Oct 30 17:56:16.133543 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.133523 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9hqsk" event={"ID":"a97de0f2-2248-44e3-bee7-9817777a261d","Type":"ContainerStarted","Data":"aa4a243c7ba6af3dddabb42fccd05303b2a068cbef89f42cb7d7616043b8ce87"} Oct 30 17:56:16.134556 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.134537 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j4bgx" event={"ID":"f2070749-ec46-433b-975e-470c734e5fe5","Type":"ContainerStarted","Data":"f6ea8465bfba56f081cad2ed99944a67a722f8d0e5f316830e04c051b6898d25"} Oct 30 17:56:16.135692 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.135614 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c" event={"ID":"e7e6e71c-794e-40ba-acef-7fddf496d546","Type":"ContainerStarted","Data":"c4cd751abe6f3d1bd40ebaeda185555b87ec00793cc51aea19aa28630130dd0b"} Oct 30 17:56:16.136565 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.136548 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" event={"ID":"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8","Type":"ContainerStarted","Data":"b63b939437dbd872c89617da541a3eb01acc58d7392962e27c89a76f21432fd8"} Oct 30 17:56:16.167424 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.167390 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dh588" podStartSLOduration=406.167379963 podStartE2EDuration="6m46.167379963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:16.166946864 +0000 UTC m=+30.756108269" watchObservedRunningTime="2026-10-30 17:56:16.167379963 +0000 UTC m=+30.756541366" Oct 30 17:56:16.208865 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.208830 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4znfd" podStartSLOduration=406.208819275 podStartE2EDuration="6m46.208819275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:16.208283741 +0000 UTC m=+30.797445145" watchObservedRunningTime="2026-10-30 17:56:16.208819275 +0000 UTC m=+30.797980678" Oct 30 17:56:16.902659 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.900926 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:16.902659 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.901384 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:16.902659 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.901834 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:16.917928 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.917909 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:56:16.918282 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.918095 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tzlmk\"" Oct 30 17:56:16.918282 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.918215 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:56:16.918282 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.918251 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dxtgs\"" Oct 30 17:56:16.925934 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.925896 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bkbt2\"" Oct 30 17:56:16.994120 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.994093 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:16.994252 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.994176 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:16.994346 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.994260 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:16.994346 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:16.994318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:16.994346 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.994324 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:56:16.994496 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.994393 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:18.994373201 +0000 UTC m=+33.583534587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : secret "router-metrics-certs-default" not found Oct 30 17:56:16.994496 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.994431 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:16.994496 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.994483 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls podName:de9d2339-d6be-4332-921d-ec5717b06699 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:18.994467272 +0000 UTC m=+33.583628667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wb2x4" (UID: "de9d2339-d6be-4332-921d-ec5717b06699") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:16.994666 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.994541 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:56:16.994666 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.994576 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert podName:f53a361f-64db-41a3-abd1-c04251831048 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:18.994565586 +0000 UTC m=+33.583726982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htdwd" (UID: "f53a361f-64db-41a3-abd1-c04251831048") : secret "networking-console-plugin-cert" not found Oct 30 17:56:16.994666 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:16.994664 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:18.994652856 +0000 UTC m=+33.583814263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : configmap references non-existent config key: service-ca.crt Oct 30 17:56:17.095452 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:17.095421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:17.095542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:17.095630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:17.095661 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:17.095780 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:17.095819 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:17.095832 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698f87866c-k4grf: secret "image-registry-tls" not found Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:17.095837 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls podName:b26b3373-a3b2-4a0d-bc89-94bd729b849e nodeName:}" failed. No retries permitted until 2026-10-30 17:56:19.095820116 +0000 UTC m=+33.684981513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls") pod "dns-default-867jv" (UID: "b26b3373-a3b2-4a0d-bc89-94bd729b849e") : secret "dns-default-metrics-tls" not found Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:17.095865 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls podName:78c06084-1bed-46c5-a576-f6129e21e062 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:19.095854813 +0000 UTC m=+33.685016199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls") pod "image-registry-698f87866c-k4grf" (UID: "78c06084-1bed-46c5-a576-f6129e21e062") : secret "image-registry-tls" not found Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:17.095920 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:17.095930 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b849f747d-p4c4m: secret "image-registry-tls" not found Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:17.095957 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls podName:1136549d-7491-452c-b396-f839668b5a19 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:19.095947067 +0000 UTC m=+33.685108451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls") pod "image-registry-5b849f747d-p4c4m" (UID: "1136549d-7491-452c-b396-f839668b5a19") : secret "image-registry-tls" not found Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:17.096039 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:56:17.096132 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:17.096090 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls podName:27d2c28b-a2b2-4293-ad25-a0e69b939b9c nodeName:}" failed. No retries permitted until 2026-10-30 17:56:19.096074476 +0000 UTC m=+33.685235872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgr98" (UID: "27d2c28b-a2b2-4293-ad25-a0e69b939b9c") : secret "samples-operator-tls" not found Oct 30 17:56:17.143400 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:17.143267 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9hqsk" event={"ID":"a97de0f2-2248-44e3-bee7-9817777a261d","Type":"ContainerStarted","Data":"e7a74b01fc00a6b52d457b3bca80403db8877779a79d1f478275351b9b3e7bb6"} Oct 30 17:56:17.188177 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:17.186979 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9hqsk" podStartSLOduration=407.186961626 podStartE2EDuration="6m47.186961626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:17.186387957 +0000 UTC m=+31.775549364" watchObservedRunningTime="2026-10-30 17:56:17.186961626 +0000 UTC m=+31.776123031" Oct 30 17:56:19.016790 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.016709 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:19.017250 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.016811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:19.017250 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.016870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:19.017250 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.016929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:19.017250 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.017068 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:56:19.017250 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.017147 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.017129351 +0000 UTC m=+37.606290750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : secret "router-metrics-certs-default" not found Oct 30 17:56:19.017571 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.017551 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:56:19.017658 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.017636 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert podName:f53a361f-64db-41a3-abd1-c04251831048 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.017612002 +0000 UTC m=+37.606773387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htdwd" (UID: "f53a361f-64db-41a3-abd1-c04251831048") : secret "networking-console-plugin-cert" not found Oct 30 17:56:19.017728 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.017704 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.017694335 +0000 UTC m=+37.606855721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : configmap references non-existent config key: service-ca.crt Oct 30 17:56:19.017785 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.017773 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:19.017837 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.017807 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls podName:de9d2339-d6be-4332-921d-ec5717b06699 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.017796429 +0000 UTC m=+37.606957825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wb2x4" (UID: "de9d2339-d6be-4332-921d-ec5717b06699") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.119849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.119890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.119918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.120014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.120191 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.120255 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls podName:27d2c28b-a2b2-4293-ad25-a0e69b939b9c nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.120241517 +0000 UTC m=+37.709402902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgr98" (UID: "27d2c28b-a2b2-4293-ad25-a0e69b939b9c") : secret "samples-operator-tls" not found Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.120617 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.120666 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls podName:b26b3373-a3b2-4a0d-bc89-94bd729b849e nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.120648797 +0000 UTC m=+37.709810182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls") pod "dns-default-867jv" (UID: "b26b3373-a3b2-4a0d-bc89-94bd729b849e") : secret "dns-default-metrics-tls" not found Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.120721 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.120730 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b849f747d-p4c4m: secret "image-registry-tls" not found Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.120764 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls podName:1136549d-7491-452c-b396-f839668b5a19 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.120753952 +0000 UTC m=+37.709915351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls") pod "image-registry-5b849f747d-p4c4m" (UID: "1136549d-7491-452c-b396-f839668b5a19") : secret "image-registry-tls" not found Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.120806 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.120813 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698f87866c-k4grf: secret "image-registry-tls" not found Oct 30 17:56:19.120857 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.120832 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls podName:78c06084-1bed-46c5-a576-f6129e21e062 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.120826084 +0000 UTC m=+37.709987469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls") pod "image-registry-698f87866c-k4grf" (UID: "78c06084-1bed-46c5-a576-f6129e21e062") : secret "image-registry-tls" not found Oct 30 17:56:19.153014 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.152981 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn" event={"ID":"bb2c63b2-119c-4f2e-a7e5-664bf9e9b999","Type":"ContainerStarted","Data":"c529e04cdcc602bc63b9ccba1611294197e6d2a9b3b6e5af25350e8ce3f19446"} Oct 30 17:56:19.725535 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.725444 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:19.725705 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.725546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:19.725705 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.725696 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:56:19.725830 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:19.725755 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs podName:6fe0d1a0-3adb-4b33-89b6-1f39fb282399 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:51.72573693 +0000 UTC m=+66.314898311 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs") pod "network-metrics-daemon-z4mc4" (UID: "6fe0d1a0-3adb-4b33-89b6-1f39fb282399") : secret "metrics-daemon-secret" not found Oct 30 17:56:19.728554 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.728526 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:19.826917 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.826879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjhv\" (UniqueName: \"kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv\") pod \"network-check-target-lzl5g\" (UID: \"92ba2348-2452-4bd2-92ba-a980fdc8a15f\") " pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:19.841370 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.841337 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjhv\" (UniqueName: \"kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv\") pod \"network-check-target-lzl5g\" (UID: \"92ba2348-2452-4bd2-92ba-a980fdc8a15f\") " pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:19.937961 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.937919 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:19.992137 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:19.992063 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:20.262566 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:20.262495 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn" podStartSLOduration=374.582702645 podStartE2EDuration="6m17.262476887s" totalImagesPullingTime="2.679774242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:19.199775849 +0000 UTC m=+33.788937256" watchObservedRunningTime="2026-10-30 17:56:20.262476887 +0000 UTC m=+34.851638293" Oct 30 17:56:20.262983 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:20.262780 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2pp59"] Oct 30 17:56:20.472085 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:20.472043 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode718a01a_548e_4499_95dc_7cbbd6ad6a12.slice/crio-1d77e2603499f2b6417e3b8ca36449ddfdac483335f5a25ef05a6c2d5fc81337 WatchSource:0}: Error finding container 1d77e2603499f2b6417e3b8ca36449ddfdac483335f5a25ef05a6c2d5fc81337: Status 404 returned error can't find the container with id 1d77e2603499f2b6417e3b8ca36449ddfdac483335f5a25ef05a6c2d5fc81337 Oct 30 17:56:21.090775 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.090748 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lzl5g"] Oct 30 17:56:21.137871 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:21.137842 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ba2348_2452_4bd2_92ba_a980fdc8a15f.slice/crio-be1aa82208a525533eaf88d066fc89da66d75b283a0ab0365e0730a053b3147d WatchSource:0}: Error finding container be1aa82208a525533eaf88d066fc89da66d75b283a0ab0365e0730a053b3147d: Status 404 returned error can't find the container with id be1aa82208a525533eaf88d066fc89da66d75b283a0ab0365e0730a053b3147d Oct 30 17:56:21.161337 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.161307 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" event={"ID":"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8","Type":"ContainerStarted","Data":"dd0ccb667b2a48be5c86edebc487e3f29ea4ba429029ac70a4fec2baf8eac170"} Oct 30 17:56:21.162994 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.162963 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2pp59" event={"ID":"e718a01a-548e-4499-95dc-7cbbd6ad6a12","Type":"ContainerStarted","Data":"1d77e2603499f2b6417e3b8ca36449ddfdac483335f5a25ef05a6c2d5fc81337"} Oct 30 17:56:21.165045 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.165023 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/0.log" Oct 30 17:56:21.165132 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.165064 2573 generic.go:356] "Generic (PLEG): container finished" podID="29549d27-df3a-48d0-8e7b-85fb883b4481" containerID="cbc650a4b0405b528ad4f1f1c3efcec7bc88b705b5f1a2afca6bd8d4856df1b2" exitCode=255 Oct 30 17:56:21.165132 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.165095 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" event={"ID":"29549d27-df3a-48d0-8e7b-85fb883b4481","Type":"ContainerDied","Data":"cbc650a4b0405b528ad4f1f1c3efcec7bc88b705b5f1a2afca6bd8d4856df1b2"} Oct 30 17:56:21.165323 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.165303 2573 scope.go:122] "RemoveContainer" containerID="cbc650a4b0405b528ad4f1f1c3efcec7bc88b705b5f1a2afca6bd8d4856df1b2" Oct 30 17:56:21.169413 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.169063 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j4bgx" event={"ID":"f2070749-ec46-433b-975e-470c734e5fe5","Type":"ContainerStarted","Data":"b208431de6eea7c9d9fd3b48eb2cea41a147fc1351ef508ce0fceed55042fd6b"} Oct 30 17:56:21.170773 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.170750 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lzl5g" event={"ID":"92ba2348-2452-4bd2-92ba-a980fdc8a15f","Type":"ContainerStarted","Data":"be1aa82208a525533eaf88d066fc89da66d75b283a0ab0365e0730a053b3147d"} Oct 30 17:56:21.207728 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.207082 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" podStartSLOduration=373.538911193 podStartE2EDuration="6m18.207065486s" totalImagesPullingTime="4.668154293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:21.205505484 +0000 UTC m=+35.794666887" watchObservedRunningTime="2026-10-30 17:56:21.207065486 +0000 UTC m=+35.796226892" Oct 30 17:56:21.753488 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.753220 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-j4bgx" podStartSLOduration=373.954182565 podStartE2EDuration="6m18.753203877s" totalImagesPullingTime="4.799021312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:21.29658773 +0000 UTC m=+35.885749135" watchObservedRunningTime="2026-10-30 17:56:21.753203877 +0000 UTC m=+36.342365278" Oct 30 17:56:21.759626 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.759573 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9gjwg"] Oct 30 17:56:21.780508 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.780480 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9gjwg"] Oct 30 17:56:21.781221 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.781191 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:21.797700 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.797652 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:56:21.797804 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.797708 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:56:21.797862 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.797811 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2dl6z\"" Oct 30 17:56:21.821350 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.821328 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fqlng-zdr42"] Oct 30 17:56:21.852315 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.852297 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fqlng-zdr42"] Oct 30 17:56:21.852505 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.852487 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:21.956027 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.955986 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b62a96-cc23-4b22-ade6-5066759bbec6-data-volume\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:21.956189 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.956089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd5pv\" (UniqueName: \"kubernetes.io/projected/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-api-access-dd5pv\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:21.956189 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.956121 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-service-ca-bundle\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:21.956189 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.956184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b62a96-cc23-4b22-ade6-5066759bbec6-crio-socket\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:21.956358 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.956256 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:21.956358 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.956297 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a65a55c-27e9-4e39-a139-53ae907e0bfd-archives-path\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:21.956358 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.956322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-trusted-ca-bundle\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:21.956358 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.956348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:21.956551 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.956374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ll6l\" (UniqueName: \"kubernetes.io/projected/1a65a55c-27e9-4e39-a139-53ae907e0bfd-kube-api-access-8ll6l\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:21.957767 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.957742 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f"] Oct 30 17:56:21.979174 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.979144 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf"] Oct 30 17:56:21.979307 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.979287 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f" Oct 30 17:56:21.996857 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.996835 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:21.997547 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.997428 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 17:56:21.997547 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:21.997524 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gbn2d\"" Oct 30 17:56:22.001352 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.001330 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f"] Oct 30 17:56:22.001442 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.001367 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf"] Oct 30 17:56:22.001496 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.001452 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf" Oct 30 17:56:22.057746 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.057721 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b62a96-cc23-4b22-ade6-5066759bbec6-crio-socket\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:22.057879 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.057820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:22.057879 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.057862 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b62a96-cc23-4b22-ade6-5066759bbec6-crio-socket\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:22.057879 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.057866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a65a55c-27e9-4e39-a139-53ae907e0bfd-archives-path\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:22.058039 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.057910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-trusted-ca-bundle\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:22.058039 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.057937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:22.058039 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.057964 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ll6l\" (UniqueName: \"kubernetes.io/projected/1a65a55c-27e9-4e39-a139-53ae907e0bfd-kube-api-access-8ll6l\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:22.058039 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.058019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b62a96-cc23-4b22-ade6-5066759bbec6-data-volume\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:22.058218 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.058100 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd5pv\" (UniqueName: \"kubernetes.io/projected/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-api-access-dd5pv\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:22.058218 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.058123 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a65a55c-27e9-4e39-a139-53ae907e0bfd-archives-path\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:22.058218 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.058130 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-service-ca-bundle\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:22.058353 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:22.058228 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:56:22.058353 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:22.058283 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:22.558256343 +0000 UTC m=+37.147417740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:56:22.058634 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.058616 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-service-ca-bundle\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:22.058892 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.058875 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b62a96-cc23-4b22-ade6-5066759bbec6-data-volume\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:22.059123 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:22.059092 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:56:22.059200 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:22.059155 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:22.559132676 +0000 UTC m=+37.148294063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : configmap "kube-rbac-proxy" not found Oct 30 17:56:22.059258 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.059206 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-trusted-ca-bundle\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:22.084031 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.084010 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ll6l\" (UniqueName: \"kubernetes.io/projected/1a65a55c-27e9-4e39-a139-53ae907e0bfd-kube-api-access-8ll6l\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:22.087104 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.087083 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd5pv\" (UniqueName: \"kubernetes.io/projected/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-api-access-dd5pv\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:22.159125 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.159090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkj95\" (UniqueName: \"kubernetes.io/projected/486056f5-ed2f-4a7f-b8c8-1130799cf4b9-kube-api-access-fkj95\") pod \"migrator-66df5d496b-qsspf\" (UID: \"486056f5-ed2f-4a7f-b8c8-1130799cf4b9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf" Oct 30 17:56:22.159366 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.159240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6pw6\" (UniqueName: \"kubernetes.io/projected/d21994f9-9c90-4711-aca3-0ebb7bbbcebe-kube-api-access-v6pw6\") pod \"migrator-66df5d496b-kz75f\" (UID: \"d21994f9-9c90-4711-aca3-0ebb7bbbcebe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f" Oct 30 17:56:22.176303 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.176275 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" event={"ID":"551adda7-a285-4f6e-9daf-b5b9f4410b48","Type":"ContainerStarted","Data":"86cd7b526512817d6867f8d107b196d3d02ecf33d2d87e9a9c97259d844a4616"} Oct 30 17:56:22.178000 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.177976 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 17:56:22.178448 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.178434 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/0.log" Oct 30 17:56:22.178526 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.178472 2573 generic.go:356] "Generic (PLEG): container finished" podID="29549d27-df3a-48d0-8e7b-85fb883b4481" containerID="c3757e8548f18241b7a39f1041137562c13252a9639c01546a9db51c789e0c49" exitCode=255 Oct 30 17:56:22.178573 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.178546 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" event={"ID":"29549d27-df3a-48d0-8e7b-85fb883b4481","Type":"ContainerDied","Data":"c3757e8548f18241b7a39f1041137562c13252a9639c01546a9db51c789e0c49"} Oct 30 17:56:22.178644 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.178631 2573 scope.go:122] "RemoveContainer" containerID="cbc650a4b0405b528ad4f1f1c3efcec7bc88b705b5f1a2afca6bd8d4856df1b2" Oct 30 17:56:22.178793 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.178775 2573 scope.go:122] "RemoveContainer" containerID="c3757e8548f18241b7a39f1041137562c13252a9639c01546a9db51c789e0c49" Oct 30 17:56:22.178982 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:22.178960 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5mn6l_openshift-console-operator(29549d27-df3a-48d0-8e7b-85fb883b4481)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" podUID="29549d27-df3a-48d0-8e7b-85fb883b4481" Oct 30 17:56:22.182917 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.182897 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vj5k4" event={"ID":"de504e55-5beb-4f9a-a08a-b4f4e1f999d6","Type":"ContainerStarted","Data":"b1950ac75e44cd8663ee288412c6f2ac0d6182d749ece804dee9c76ca97d18a9"} Oct 30 17:56:22.184916 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.184891 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lzl5g" event={"ID":"92ba2348-2452-4bd2-92ba-a980fdc8a15f","Type":"ContainerStarted","Data":"4fa21681950d3ea30102b20f4e833d22c57c55941df61195030613f9e946f2a4"} Oct 30 17:56:22.185042 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.185033 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:22.186616 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.186572 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c" event={"ID":"e7e6e71c-794e-40ba-acef-7fddf496d546","Type":"ContainerStarted","Data":"df26d4537f7f5b4c2376e2e378ebb413dff647c4d6425df50f880df2c2e2b9c0"} Oct 30 17:56:22.201152 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.201131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:22.219770 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.219725 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" podStartSLOduration=374.077750325 podStartE2EDuration="6m19.219713404s" totalImagesPullingTime="5.141963079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:22.219080715 +0000 UTC m=+36.808242123" watchObservedRunningTime="2026-10-30 17:56:22.219713404 +0000 UTC m=+36.808874807" Oct 30 17:56:22.260828 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.260524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkj95\" (UniqueName: \"kubernetes.io/projected/486056f5-ed2f-4a7f-b8c8-1130799cf4b9-kube-api-access-fkj95\") pod \"migrator-66df5d496b-qsspf\" (UID: \"486056f5-ed2f-4a7f-b8c8-1130799cf4b9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf" Oct 30 17:56:22.260828 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.260701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6pw6\" (UniqueName: \"kubernetes.io/projected/d21994f9-9c90-4711-aca3-0ebb7bbbcebe-kube-api-access-v6pw6\") pod \"migrator-66df5d496b-kz75f\" (UID: \"d21994f9-9c90-4711-aca3-0ebb7bbbcebe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f" Oct 30 17:56:22.295758 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.295714 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkj95\" (UniqueName: \"kubernetes.io/projected/486056f5-ed2f-4a7f-b8c8-1130799cf4b9-kube-api-access-fkj95\") pod \"migrator-66df5d496b-qsspf\" (UID: \"486056f5-ed2f-4a7f-b8c8-1130799cf4b9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf" Oct 30 17:56:22.296124 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.296092 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6pw6\" (UniqueName: \"kubernetes.io/projected/d21994f9-9c90-4711-aca3-0ebb7bbbcebe-kube-api-access-v6pw6\") pod \"migrator-66df5d496b-kz75f\" (UID: \"d21994f9-9c90-4711-aca3-0ebb7bbbcebe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f" Oct 30 17:56:22.308570 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.308524 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f" Oct 30 17:56:22.314214 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.314127 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vj5k4" podStartSLOduration=2.251904412 podStartE2EDuration="7.314111526s" totalImagesPullingTime="5.062207114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:22.312907686 +0000 UTC m=+36.902069099" watchObservedRunningTime="2026-10-30 17:56:22.314111526 +0000 UTC m=+36.903272932" Oct 30 17:56:22.334117 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.334094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf" Oct 30 17:56:22.364907 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.364854 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lzl5g" podStartSLOduration=35.7712412 podStartE2EDuration="36.364835066s" totalImagesPullingTime="593.593866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:22.362861091 +0000 UTC m=+36.952022498" watchObservedRunningTime="2026-10-30 17:56:22.364835066 +0000 UTC m=+36.953996471" Oct 30 17:56:22.561582 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.561534 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c" podStartSLOduration=374.121208038 podStartE2EDuration="6m19.561518309s" totalImagesPullingTime="5.440310271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:22.421774544 +0000 UTC m=+37.010935962" watchObservedRunningTime="2026-10-30 17:56:22.561518309 +0000 UTC m=+37.150679714" Oct 30 17:56:22.562775 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.562742 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fqlng-zdr42"] Oct 30 17:56:22.563432 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:22.563400 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a65a55c_27e9_4e39_a139_53ae907e0bfd.slice/crio-cc7303134e85a69c59e996f3b5948a6cb5fc57ae411f334fbf7dc659bd3ecb2d WatchSource:0}: Error finding container cc7303134e85a69c59e996f3b5948a6cb5fc57ae411f334fbf7dc659bd3ecb2d: Status 404 returned error can't find the container with id cc7303134e85a69c59e996f3b5948a6cb5fc57ae411f334fbf7dc659bd3ecb2d Oct 30 17:56:22.565185 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.565163 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:22.565283 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.565216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:22.565403 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:22.565338 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:56:22.565403 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:22.565341 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:56:22.565403 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:22.565392 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.565375726 +0000 UTC m=+38.154537126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : configmap "kube-rbac-proxy" not found Oct 30 17:56:22.565555 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:22.565410 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.565400692 +0000 UTC m=+38.154562074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:56:22.648886 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.648808 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f"] Oct 30 17:56:22.650956 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:22.650928 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21994f9_9c90_4711_aca3_0ebb7bbbcebe.slice/crio-c8430f93c69c9ca43f3fe2725bd858512bf0e99e2459f2a05f9088a40e383430 WatchSource:0}: Error finding container c8430f93c69c9ca43f3fe2725bd858512bf0e99e2459f2a05f9088a40e383430: Status 404 returned error can't find the container with id c8430f93c69c9ca43f3fe2725bd858512bf0e99e2459f2a05f9088a40e383430 Oct 30 17:56:22.684704 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.684674 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf"] Oct 30 17:56:22.686018 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:22.685987 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486056f5_ed2f_4a7f_b8c8_1130799cf4b9.slice/crio-80f17718bf4c4d5569bbdbc175f558fe131456e7a301666a7a7be10be782ce51 WatchSource:0}: Error finding container 80f17718bf4c4d5569bbdbc175f558fe131456e7a301666a7a7be10be782ce51: Status 404 returned error can't find the container with id 80f17718bf4c4d5569bbdbc175f558fe131456e7a301666a7a7be10be782ce51 Oct 30 17:56:22.940859 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.940793 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lpb2n"] Oct 30 17:56:22.969751 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.969716 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lpb2n"] Oct 30 17:56:22.969891 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.969775 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:22.986409 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.986311 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 17:56:22.986409 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.986360 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 17:56:22.986584 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.986446 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pjbxh\"" Oct 30 17:56:22.986584 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.986453 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 17:56:22.986822 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:22.986797 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 17:56:23.070217 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.070186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:23.070341 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.070228 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4dg9\" (UniqueName: \"kubernetes.io/projected/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-kube-api-access-p4dg9\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:23.070341 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.070280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:23.070341 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.070311 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:56:23.070341 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.070325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:23.070468 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.070413 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:23.070468 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.070426 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.070409582 +0000 UTC m=+45.659570984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : configmap references non-existent config key: service-ca.crt Oct 30 17:56:23.070468 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.070446 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert podName:f53a361f-64db-41a3-abd1-c04251831048 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.070437259 +0000 UTC m=+45.659598642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htdwd" (UID: "f53a361f-64db-41a3-abd1-c04251831048") : secret "networking-console-plugin-cert" not found Oct 30 17:56:23.070468 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.070459 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls podName:de9d2339-d6be-4332-921d-ec5717b06699 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.070451722 +0000 UTC m=+45.659613103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wb2x4" (UID: "de9d2339-d6be-4332-921d-ec5717b06699") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:23.070589 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.070486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-key\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:23.070589 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.070531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:23.070664 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.070633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-cabundle\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:23.070664 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.070659 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:56:23.070721 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.070696 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.070684274 +0000 UTC m=+45.659845656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : secret "router-metrics-certs-default" not found Oct 30 17:56:23.171297 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.171272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4dg9\" (UniqueName: \"kubernetes.io/projected/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-kube-api-access-p4dg9\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:23.171433 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.171304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:23.171433 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.171325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:23.171433 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.171344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:23.171433 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.171410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-key\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:23.171644 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.171437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:23.171644 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.171470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-cabundle\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:23.172081 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.172064 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-cabundle\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:23.172333 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.172319 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:56:23.172386 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.172377 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls podName:b26b3373-a3b2-4a0d-bc89-94bd729b849e nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.172363861 +0000 UTC m=+45.761525261 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls") pod "dns-default-867jv" (UID: "b26b3373-a3b2-4a0d-bc89-94bd729b849e") : secret "dns-default-metrics-tls" not found Oct 30 17:56:23.172650 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.172636 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:56:23.172710 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.172653 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b849f747d-p4c4m: secret "image-registry-tls" not found Oct 30 17:56:23.172710 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.172694 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls podName:1136549d-7491-452c-b396-f839668b5a19 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.172680153 +0000 UTC m=+45.761841549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls") pod "image-registry-5b849f747d-p4c4m" (UID: "1136549d-7491-452c-b396-f839668b5a19") : secret "image-registry-tls" not found Oct 30 17:56:23.172807 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.172752 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:56:23.172807 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.172783 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls podName:27d2c28b-a2b2-4293-ad25-a0e69b939b9c nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.172772322 +0000 UTC m=+45.761933710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgr98" (UID: "27d2c28b-a2b2-4293-ad25-a0e69b939b9c") : secret "samples-operator-tls" not found Oct 30 17:56:23.172906 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.172837 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:56:23.172906 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.172846 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698f87866c-k4grf: secret "image-registry-tls" not found Oct 30 17:56:23.172906 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.172875 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls podName:78c06084-1bed-46c5-a576-f6129e21e062 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.172866325 +0000 UTC m=+45.762027709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls") pod "image-registry-698f87866c-k4grf" (UID: "78c06084-1bed-46c5-a576-f6129e21e062") : secret "image-registry-tls" not found Oct 30 17:56:23.175570 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.175544 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-key\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:23.192562 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.192512 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 17:56:23.192914 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.192876 2573 scope.go:122] "RemoveContainer" containerID="c3757e8548f18241b7a39f1041137562c13252a9639c01546a9db51c789e0c49" Oct 30 17:56:23.193107 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.193086 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5mn6l_openshift-console-operator(29549d27-df3a-48d0-8e7b-85fb883b4481)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" podUID="29549d27-df3a-48d0-8e7b-85fb883b4481" Oct 30 17:56:23.194040 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.194021 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf" event={"ID":"486056f5-ed2f-4a7f-b8c8-1130799cf4b9","Type":"ContainerStarted","Data":"80f17718bf4c4d5569bbdbc175f558fe131456e7a301666a7a7be10be782ce51"} Oct 30 17:56:23.196168 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.196147 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fqlng-zdr42" event={"ID":"1a65a55c-27e9-4e39-a139-53ae907e0bfd","Type":"ContainerStarted","Data":"e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822"} Oct 30 17:56:23.196262 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.196187 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fqlng-zdr42" event={"ID":"1a65a55c-27e9-4e39-a139-53ae907e0bfd","Type":"ContainerStarted","Data":"cc7303134e85a69c59e996f3b5948a6cb5fc57ae411f334fbf7dc659bd3ecb2d"} Oct 30 17:56:23.197285 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.197267 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4dg9\" (UniqueName: \"kubernetes.io/projected/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-kube-api-access-p4dg9\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:23.201284 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.199699 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f" event={"ID":"d21994f9-9c90-4711-aca3-0ebb7bbbcebe","Type":"ContainerStarted","Data":"c8430f93c69c9ca43f3fe2725bd858512bf0e99e2459f2a05f9088a40e383430"} Oct 30 17:56:23.283148 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.283088 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fqlng-zdr42" podStartSLOduration=2.283071399 podStartE2EDuration="2.283071399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:23.281872348 +0000 UTC m=+37.871033753" watchObservedRunningTime="2026-10-30 17:56:23.283071399 +0000 UTC m=+37.872232803" Oct 30 17:56:23.296790 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.296610 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:23.577196 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.577159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:23.577389 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.577228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:23.577389 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.577345 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:56:23.577389 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.577372 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:56:23.577563 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.577409 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:25.577389337 +0000 UTC m=+40.166550734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : configmap "kube-rbac-proxy" not found Oct 30 17:56:23.577563 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:23.577438 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:25.577420324 +0000 UTC m=+40.166581720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:56:23.618226 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:23.618204 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lpb2n"] Oct 30 17:56:23.619153 ip-10-0-133-62 kubenswrapper[2573]: W1030 17:56:23.619129 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bcd372_b9ea_4a6c_b9ce_b309d04b3ed5.slice/crio-1bbc517a66512bc6168c017f3267efca9de6c8a80eeae51d7ce1ee3be77455e1 WatchSource:0}: Error finding container 1bbc517a66512bc6168c017f3267efca9de6c8a80eeae51d7ce1ee3be77455e1: Status 404 returned error can't find the container with id 1bbc517a66512bc6168c017f3267efca9de6c8a80eeae51d7ce1ee3be77455e1 Oct 30 17:56:24.204220 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:24.204186 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" event={"ID":"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5","Type":"ContainerStarted","Data":"7d7d78c2186b818c889aec36d313fd03d6fd1f90e224069d3963cc5f54afa16d"} Oct 30 17:56:24.204776 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:24.204234 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" event={"ID":"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5","Type":"ContainerStarted","Data":"1bbc517a66512bc6168c017f3267efca9de6c8a80eeae51d7ce1ee3be77455e1"} Oct 30 17:56:24.248910 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:24.248873 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" podStartSLOduration=2.248861032 podStartE2EDuration="2.248861032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:24.247978008 +0000 UTC m=+38.837139412" watchObservedRunningTime="2026-10-30 17:56:24.248861032 +0000 UTC m=+38.838022435" Oct 30 17:56:24.875877 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:24.875843 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-94ttz_f1043899-80a3-47a9-9d1c-0421cd68df00/dns-node-resolver/0.log" Oct 30 17:56:25.210641 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:25.210253 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf" event={"ID":"486056f5-ed2f-4a7f-b8c8-1130799cf4b9","Type":"ContainerStarted","Data":"f421a5afdcaec046cc497e0d4c87918a67b1107c1c36d8d0f4e93f9f9977a927"} Oct 30 17:56:25.216235 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:25.216205 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f" event={"ID":"d21994f9-9c90-4711-aca3-0ebb7bbbcebe","Type":"ContainerStarted","Data":"7b7f892e6e6431f63a1a22a5fa62429ea8c72cc14d373ff5f716e04734126019"} Oct 30 17:56:25.254962 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:25.254052 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf" podStartSLOduration=2.287320695 podStartE2EDuration="4.254036158s" totalImagesPullingTime="1.966715463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:25.252731909 +0000 UTC m=+39.841893310" watchObservedRunningTime="2026-10-30 17:56:25.254036158 +0000 UTC m=+39.843197565" Oct 30 17:56:25.296103 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:25.296041 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f" podStartSLOduration=2.490746281 podStartE2EDuration="4.296026403s" totalImagesPullingTime="1.805280122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:25.294434418 +0000 UTC m=+39.883595822" watchObservedRunningTime="2026-10-30 17:56:25.296026403 +0000 UTC m=+39.885187810" Oct 30 17:56:25.605415 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:25.605386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:25.605641 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:25.605443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:25.605641 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:25.605518 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:56:25.605641 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:25.605534 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:56:25.605641 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:25.605611 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:29.605575498 +0000 UTC m=+44.194736883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:56:25.605879 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:25.605649 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:29.605637059 +0000 UTC m=+44.194798443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : configmap "kube-rbac-proxy" not found Oct 30 17:56:25.667879 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:25.667840 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:25.668033 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:25.667892 2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:25.668291 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:25.668279 2573 scope.go:122] "RemoveContainer" containerID="c3757e8548f18241b7a39f1041137562c13252a9639c01546a9db51c789e0c49" Oct 30 17:56:25.668499 ip-10-0-133-62 kubenswrapper[2573]: E1030 17:56:25.668480 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5mn6l_openshift-console-operator(29549d27-df3a-48d0-8e7b-85fb883b4481)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" podUID="29549d27-df3a-48d0-8e7b-85fb883b4481" Oct 30 17:56:25.875437 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:25.875359 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l67qw_f4572363-4113-4e99-a34a-1b33a4cc08ff/node-ca/0.log" Oct 30 17:56:26.676492 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:26.676467 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vj5k4_de504e55-5beb-4f9a-a08a-b4f4e1f999d6/serve-healthcheck-canary/0.log" Oct 30 17:56:26.876672 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:26.876609 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-kz75f_d21994f9-9c90-4711-aca3-0ebb7bbbcebe/migrator/0.log" Oct 30 17:56:26.941493 ip-10-0-133-62 kubenswrapper[2573]: I1030 17:56:26.941405 2573 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:56:26.941580 ip-10-0-133-62 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 17:56:26.952937 ip-10-0-133-62 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 17:56:26.953099 ip-10-0-133-62 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 17:56:26.953145 ip-10-0-133-62 systemd[1]: kubelet.service: Consumed 3.124s CPU time. Oct 30 17:56:26.968184 ip-10-0-133-62 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:56:27.053196 ip-10-0-133-62 kubenswrapper[5014]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:56:27.053196 ip-10-0-133-62 kubenswrapper[5014]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:56:27.053196 ip-10-0-133-62 kubenswrapper[5014]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:56:27.053196 ip-10-0-133-62 kubenswrapper[5014]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:56:27.055201 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055136 5014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:27.055201 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055203 5014 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055208 5014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055211 5014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055214 5014 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055217 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055220 5014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055222 5014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055226 5014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055228 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055231 5014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055234 5014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055236 5014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055239 5014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055242 5014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055244 5014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055247 5014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055250 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055253 5014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055255 5014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055263 5014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:27.055277 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055266 5014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055269 5014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055272 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055274 5014 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055277 5014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055280 5014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055301 5014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055305 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055309 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055312 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055315 5014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055317 5014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055321 5014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055324 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055326 5014 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055329 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055331 5014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055334 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055336 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:27.055805 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055339 5014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055341 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055344 5014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055346 5014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055349 5014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055357 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055360 5014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055363 5014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055365 5014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055368 5014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055371 5014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055373 5014 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055376 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055378 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055382 5014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055385 5014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055387 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055390 5014 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055392 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:27.056316 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055397 5014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055401 5014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055405 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055408 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055410 5014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055413 5014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055415 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055418 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055421 5014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055424 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055426 5014 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055429 5014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055431 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055434 5014 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055437 5014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055439 5014 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055442 5014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055444 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055447 5014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:27.056907 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055449 5014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055452 5014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055454 5014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055457 5014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055461 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055463 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055465 5014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055468 5014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055471 5014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055474 5014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055476 5014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055479 5014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055482 5014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055484 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055487 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055489 5014 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055492 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055494 5014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055497 5014 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055500 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:27.057494 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055503 5014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055506 5014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055508 5014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055511 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055513 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055516 5014 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055518 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.055522 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056273 5014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056280 5014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056284 5014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056287 5014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056290 5014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056293 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056295 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056298 5014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056301 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056304 5014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056307 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056311 5014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:27.058090 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056313 5014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056316 5014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056319 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056321 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056324 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056327 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056329 5014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056333 5014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056335 5014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056338 5014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056340 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056348 5014 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056351 5014 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056353 5014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056356 5014 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056359 5014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056361 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056364 5014 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056366 5014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056369 5014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:27.058650 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056371 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056374 5014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056376 5014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056379 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056381 5014 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056384 5014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056386 5014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056389 5014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056391 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056394 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056397 5014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056408 5014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056412 5014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056415 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056418 5014 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056420 5014 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056423 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056426 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056428 5014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:27.059832 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056430 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056433 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056436 5014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056438 5014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056442 5014 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056445 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056448 5014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056450 5014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056453 5014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056455 5014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056458 5014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056460 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056463 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056466 5014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056468 5014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056471 5014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056474 5014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056479 5014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056482 5014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:27.060483 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056486 5014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056488 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056491 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056494 5014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056496 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056500 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056503 5014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056505 5014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056509 5014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056512 5014 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056515 5014 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056518 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056523 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056525 5014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056528 5014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056530 5014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056533 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056537 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056539 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:27.061004 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056542 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056544 5014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056547 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056550 5014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056552 5014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056555 5014 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056557 5014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056560 5014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056562 5014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056565 5014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056568 5014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056571 5014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056573 5014 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056576 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056578 5014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056581 5014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.056584 5014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057223 5014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057235 5014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057238 5014 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:27.061514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057241 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057244 5014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057247 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057250 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057253 5014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057255 5014 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057258 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057260 5014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057265 5014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057269 5014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057275 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057278 5014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057281 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057284 5014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057286 5014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057289 5014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057292 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057295 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057297 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:27.062148 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057300 5014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057303 5014 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057305 5014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057310 5014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057312 5014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057315 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057317 5014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057320 5014 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057323 5014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057325 5014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057328 5014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057330 5014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057333 5014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057336 5014 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057339 5014 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057341 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057344 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057347 5014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057349 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057352 5014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:27.063020 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057354 5014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057357 5014 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057359 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057363 5014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057366 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057368 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057371 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057373 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057376 5014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057378 5014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057381 5014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057383 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057386 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057388 5014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057391 5014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057395 5014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057398 5014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057400 5014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057403 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057405 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:27.063933 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057408 5014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057410 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057413 5014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057415 5014 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057417 5014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057420 5014 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057423 5014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057426 5014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057428 5014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057431 5014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057433 5014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057435 5014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057438 5014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057440 5014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057443 5014 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057446 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057449 5014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057452 5014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057454 5014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:27.064831 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057458 5014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057461 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057464 5014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057467 5014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057469 5014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057472 5014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057474 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057477 5014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057481 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057483 5014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057486 5014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057488 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057491 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057493 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057496 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057498 5014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057501 5014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057503 5014 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057506 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:27.065719 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057510 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057513 5014 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057516 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057518 5014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057521 5014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.057523 5014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057658 5014 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057667 5014 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057673 5014 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057678 5014 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057683 5014 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057686 5014 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057693 5014 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057697 5014 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057700 5014 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057704 5014 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057707 5014 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057710 5014 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057713 5014 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057716 5014 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057719 5014 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057723 5014 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057726 5014 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:56:27.066584 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057729 5014 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057733 5014 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057736 5014 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057739 5014 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057742 5014 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057745 5014 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057749 5014 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057752 5014 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057755 5014 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057758 5014 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057761 5014 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057764 5014 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057767 5014 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057770 5014 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057773 5014 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057778 5014 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057781 5014 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057783 5014 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057786 5014 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057790 5014 flags.go:64] FLAG: --enable-server="true" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057792 5014 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057796 5014 flags.go:64] FLAG: --event-burst="100" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057799 5014 flags.go:64] FLAG: --event-qps="50" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057802 5014 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:56:27.067654 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057805 5014 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057808 5014 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057812 5014 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057815 5014 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057818 5014 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057821 5014 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057825 5014 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057828 5014 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057831 5014 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057834 5014 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057836 5014 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057839 5014 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057842 5014 flags.go:64] FLAG: --feature-gates="" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057845 5014 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057849 5014 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057852 5014 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057855 5014 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057857 5014 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057861 5014 flags.go:64] FLAG: --help="false" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057864 5014 flags.go:64] FLAG: --hostname-override="ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057867 5014 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057870 5014 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057873 5014 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057876 5014 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:56:27.068757 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057880 5014 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057883 5014 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057885 5014 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057888 5014 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057891 5014 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057893 5014 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057896 5014 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057899 5014 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057902 5014 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057905 5014 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057908 5014 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057910 5014 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057913 5014 flags.go:64] FLAG: --lock-file="" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057916 5014 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057920 5014 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057923 5014 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057929 5014 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057932 5014 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057935 5014 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057954 5014 flags.go:64] FLAG: --logging-format="text" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057959 5014 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057963 5014 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057966 5014 flags.go:64] FLAG: --manifest-url="" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057969 5014 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057973 5014 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:56:27.069828 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057976 5014 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057980 5014 flags.go:64] FLAG: --max-pods="110" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057984 5014 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057988 5014 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057991 5014 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057994 5014 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.057997 5014 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058000 5014 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058003 5014 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058010 5014 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058013 5014 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058016 5014 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058019 5014 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058022 5014 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058024 5014 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058027 5014 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058030 5014 flags.go:64] FLAG: --port="10250" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058033 5014 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058036 5014 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fc01e439f55d70eb" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058039 5014 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058042 5014 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058046 5014 flags.go:64] FLAG: --register-node="true" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058049 5014 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058052 5014 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058055 5014 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058058 5014 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:56:27.070962 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058061 5014 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058064 5014 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058067 5014 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058070 5014 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058073 5014 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058076 5014 flags.go:64] FLAG: --runonce="false" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058079 5014 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058082 5014 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058086 5014 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058089 5014 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058094 5014 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058097 5014 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058100 5014 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058104 5014 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058107 5014 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058110 5014 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058113 5014 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058116 5014 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058119 5014 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058122 5014 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058124 5014 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058129 5014 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058132 5014 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058135 5014 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058138 5014 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:56:27.071915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058141 5014 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058144 5014 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058147 5014 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058150 5014 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058153 5014 flags.go:64] FLAG: --v="2" Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058157 5014 flags.go:64] FLAG: --version="false" Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058161 5014 flags.go:64] FLAG: --vmodule="" Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058164 5014 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.058168 5014 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058295 5014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058299 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058302 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058305 5014 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058309 5014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058313 5014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058317 5014 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058319 5014 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058322 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058325 5014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058328 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058331 5014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:27.072869 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058333 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058335 5014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058338 5014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058341 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058343 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058346 5014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058348 5014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058351 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058353 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058356 5014 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058358 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058361 5014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058363 5014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058366 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058369 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058371 5014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058374 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058376 5014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058379 5014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:27.073420 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058381 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058384 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058386 5014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058389 5014 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058391 5014 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058394 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058396 5014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058401 5014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058404 5014 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058406 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058409 5014 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058412 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058415 5014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058417 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058420 5014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058423 5014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058425 5014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058428 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058430 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:27.073896 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058432 5014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058435 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058437 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058440 5014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058442 5014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058445 5014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058447 5014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058450 5014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058452 5014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058455 5014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058457 5014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058460 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058462 5014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058465 5014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058469 5014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058473 5014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058476 5014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058479 5014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058481 5014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:27.074514 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058484 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058487 5014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.058490 5014 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059025 5014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059054 5014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059062 5014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059070 5014 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059077 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059082 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059086 5014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059091 5014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059097 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059102 5014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059107 5014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059112 5014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059122 5014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059127 5014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059131 5014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059136 5014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059140 5014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:27.075088 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059145 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059149 5014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059153 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059157 5014 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059162 5014 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059166 5014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059170 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059179 5014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059183 5014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059187 5014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059191 5014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059195 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059200 5014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059206 5014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059211 5014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059216 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:27.075643 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059220 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.059284 5014 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059399 5014 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059409 5014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059417 5014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059422 5014 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059427 5014 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059432 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059437 5014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059446 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059450 5014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059454 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059458 5014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059462 5014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059469 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059473 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:27.076039 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059477 5014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059481 5014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059486 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059491 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059494 5014 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059498 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059507 5014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059511 5014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059515 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059519 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059523 5014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059527 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059531 5014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059535 5014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059541 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059545 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059549 5014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059553 5014 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059558 5014 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:27.076446 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059581 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059609 5014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059614 5014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059619 5014 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059624 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059628 5014 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059632 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059636 5014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059641 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059645 5014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059649 5014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059655 5014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059659 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059668 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059672 5014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059676 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059681 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059684 5014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059689 5014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059692 5014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:27.076964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059696 5014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059701 5014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059705 5014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059709 5014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059714 5014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059718 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059732 5014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059737 5014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059741 5014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059746 5014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059749 5014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059755 5014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059758 5014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059761 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059764 5014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059767 5014 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059770 5014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059776 5014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059779 5014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:27.077490 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059782 5014 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059786 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059790 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059795 5014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059801 5014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059805 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059808 5014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059811 5014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059814 5014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059817 5014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059822 5014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059825 5014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059827 5014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059831 5014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059834 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059836 5014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059861 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059879 5014 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059882 5014 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059991 5014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:27.078217 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.059999 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060002 5014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060005 5014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060008 5014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060011 5014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060013 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060016 5014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060018 5014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060021 5014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060023 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060026 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060028 5014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060031 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.060034 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.064576 5014 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.064611 5014 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.064636 5014 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.064644 5014 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064748 5014 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:27.078972 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064757 5014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064762 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064767 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064772 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064777 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064781 5014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064786 5014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064790 5014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064795 5014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064800 5014 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064804 5014 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064809 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064813 5014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064820 5014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064827 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064832 5014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064838 5014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064845 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064850 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:27.079504 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064855 5014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064860 5014 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064864 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064869 5014 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064873 5014 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064878 5014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064882 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064887 5014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064891 5014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064896 5014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064900 5014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064905 5014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064909 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064913 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064918 5014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064922 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064927 5014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064931 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064936 5014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064940 5014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:27.079997 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064944 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064949 5014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064953 5014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064958 5014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064962 5014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064967 5014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064971 5014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064975 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064979 5014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064983 5014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064988 5014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064992 5014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.064996 5014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065001 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065006 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065010 5014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065014 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065018 5014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065023 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:27.080752 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065027 5014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065031 5014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065037 5014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065041 5014 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065046 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065050 5014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065055 5014 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065060 5014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065064 5014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065068 5014 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065072 5014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065076 5014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065081 5014 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065086 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065090 5014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065094 5014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065099 5014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065103 5014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065107 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065111 5014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:27.081377 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065115 5014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065119 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065124 5014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065128 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065133 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065137 5014 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065141 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065145 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065150 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065155 5014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065159 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065163 5014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065167 5014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065172 5014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065176 5014 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065180 5014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065185 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065189 5014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065194 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:27.081968 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065200 5014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065204 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065208 5014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065212 5014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065216 5014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065221 5014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065225 5014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065230 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.065270 5014 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065496 5014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065504 5014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065509 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065513 5014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065518 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065522 5014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065552 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:27.082416 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065559 5014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065564 5014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065569 5014 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065574 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065579 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065583 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065587 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065617 5014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065621 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065626 5014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065631 5014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065635 5014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065640 5014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065644 5014 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065651 5014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065658 5014 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065663 5014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065669 5014 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065673 5014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:27.082828 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065677 5014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065682 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065686 5014 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065690 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065694 5014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065699 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065703 5014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065708 5014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065712 5014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065716 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065723 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065727 5014 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065731 5014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065735 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065739 5014 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065743 5014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065748 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065752 5014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065757 5014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065761 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:27.083278 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065766 5014 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065770 5014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065775 5014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065779 5014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065785 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065790 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065794 5014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065799 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065803 5014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065807 5014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065812 5014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065818 5014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065823 5014 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065827 5014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065831 5014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065835 5014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065840 5014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065844 5014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065849 5014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065853 5014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:27.083913 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065857 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065861 5014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065868 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065872 5014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065876 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065880 5014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065885 5014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065890 5014 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065894 5014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065898 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065903 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065907 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065912 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065916 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065920 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065924 5014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065930 5014 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065935 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065939 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065944 5014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:27.084428 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065948 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065952 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065957 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065961 5014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065966 5014 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065971 5014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065975 5014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065980 5014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065984 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065988 5014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065993 5014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.065997 5014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066001 5014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066008 5014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066016 5014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066021 5014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066027 5014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066031 5014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066037 5014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:27.084935 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066041 5014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.066083 5014 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066140 5014 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066146 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066151 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066155 5014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066160 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066164 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066168 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066173 5014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066179 5014 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066183 5014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066187 5014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066192 5014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066196 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066200 5014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:27.085392 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066205 5014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066209 5014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066214 5014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066220 5014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066225 5014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066229 5014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066233 5014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066238 5014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066242 5014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066246 5014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066251 5014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066257 5014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066261 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066265 5014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066269 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066274 5014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066278 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066283 5014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066287 5014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:27.085842 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066291 5014 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066295 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066300 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066304 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066309 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066313 5014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066317 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066323 5014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066328 5014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066332 5014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066336 5014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066340 5014 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066347 5014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066353 5014 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066358 5014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066363 5014 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066368 5014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066373 5014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066377 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:27.086298 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066381 5014 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066386 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066390 5014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066394 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066398 5014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066404 5014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066409 5014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066413 5014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066417 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066422 5014 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066427 5014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066431 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066436 5014 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066440 5014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066445 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066449 5014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066453 5014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066457 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066462 5014 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:27.087003 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066466 5014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066472 5014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066476 5014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066480 5014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066485 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066489 5014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066493 5014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066497 5014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066502 5014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066507 5014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066511 5014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066516 5014 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066520 5014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066525 5014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066529 5014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066533 5014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066537 5014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066541 5014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066547 5014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066552 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:27.087861 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066556 5014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066561 5014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066565 5014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066569 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066573 5014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066577 5014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066584 5014 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066588 5014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066609 5014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066613 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066617 5014 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066621 5014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066625 5014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066630 5014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:27.066634 5014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.066715 5014 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.068233 5014 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.068304 5014 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.068608 5014 server.go:1014] "Starting client certificate rotation" Oct 30 17:56:27.088744 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.068735 5014 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:56:27.089652 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.068792 5014 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:50:46 +0000 UTC" deadline="2028-04-04 06:18:42.542333455 +0000 UTC" Oct 30 17:56:27.089652 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.068819 5014 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12516h22m15.473516571s" Oct 30 17:56:27.089652 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.069336 5014 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:56:27.089652 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.071149 5014 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:56:27.089652 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.072360 5014 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:56:27.089652 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.074166 5014 log.go:25] "Validated CRI v1 image API" Oct 30 17:56:27.089652 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.074973 5014 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:56:27.089652 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.077328 5014 fs.go:135] Filesystem UUIDs: map[33969049-38b9-4442-a2f9-c8530e868690:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9bc7bfbd-5451-40d5-a0a7-1721da0f8336:/dev/nvme0n1p4] Oct 30 17:56:27.090037 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.077347 5014 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c15a7e453d497f6a249ff157b6b3a54ba12fa006933cca34c9fe902edc35786/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c15a7e453d497f6a249ff157b6b3a54ba12fa006933cca34c9fe902edc35786/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12e41e5e05545f170de90c819a3692a0a7b2a6bf52372ecc4a7c0e7b7da34845/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12e41e5e05545f170de90c819a3692a0a7b2a6bf52372ecc4a7c0e7b7da34845/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bbc517a66512bc6168c017f3267efca9de6c8a80eeae51d7ce1ee3be77455e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bbc517a66512bc6168c017f3267efca9de6c8a80eeae51d7ce1ee3be77455e1/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d77e2603499f2b6417e3b8ca36449ddfdac483335f5a25ef05a6c2d5fc81337/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d77e2603499f2b6417e3b8ca36449ddfdac483335f5a25ef05a6c2d5fc81337/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e84fbad55f885a2bb894c95b6d4b4280cd7e6dadb02873fe5aa6d4d447e6201/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e84fbad55f885a2bb894c95b6d4b4280cd7e6dadb02873fe5aa6d4d447e6201/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6dc72b7434eeea4524819dde3b67eb31988432a1bd628c63952b50316e0946bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6dc72b7434eeea4524819dde3b67eb31988432a1bd628c63952b50316e0946bc/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6eeef5b820286f6ffc3e108c52811f157c97b0effc4a4484ad24bcf1a0d8a2fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6eeef5b820286f6ffc3e108c52811f157c97b0effc4a4484ad24bcf1a0d8a2fd/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7617edbac47192048c6e4eaf4cdfbcf583584596e790eb4f45e2c0f6f8f3a363/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7617edbac47192048c6e4eaf4cdfbcf583584596e790eb4f45e2c0f6f8f3a363/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80ed2bfab374e0140e9c9253103658cb87017bfb519929d673b82c54d75efd4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80ed2bfab374e0140e9c9253103658cb87017bfb519929d673b82c54d75efd4e/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80f17718bf4c4d5569bbdbc175f558fe131456e7a301666a7a7be10be782ce51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80f17718bf4c4d5569bbdbc175f558fe131456e7a301666a7a7be10be782ce51/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ce825231d61103e72d15ad82becc935dfe0cbea92fe490337dd2ad2ce9c25ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ce825231d61103e72d15ad82becc935dfe0cbea92fe490337dd2ad2ce9c25ac/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d02664274cfb9bf9de77c00f6954e5b228ca3ba9ff4e19497d60a0c04a14a91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d02664274cfb9bf9de77c00f6954e5b228ca3ba9ff4e19497d60a0c04a14a91/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e2650d8ec1d7c969cf0fee983ebe7c48d0543299c11bcf27c6b7fe631b9ad15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e2650d8ec1d7c969cf0fee983ebe7c48d0543299c11bcf27c6b7fe631b9ad15/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa4a243c7ba6af3dddabb42fccd05303b2a068cbef89f42cb7d7616043b8ce87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa4a243c7ba6af3dddabb42fccd05303b2a068cbef89f42cb7d7616043b8ce87/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1f52a68261f62e38dd998161ceb6188a65e071c483afa4a3b43b1eb4f3ed105/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1f52a68261f62e38dd998161ceb6188a65e071c483afa4a3b43b1eb4f3ed105/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b63b939437dbd872c89617da541a3eb01acc58d7392962e27c89a76f21432fd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b63b939437dbd872c89617da541a3eb01acc58d7392962e27c89a76f21432fd8/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd720384799e29bce3b473e206fdb2afa4f4010ee72a5faf40f105472a4f29ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd720384799e29bce3b473e206fdb2afa4f4010ee72a5faf40f105472a4f29ea/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be1aa82208a525533eaf88d066fc89da66d75b283a0ab0365e0730a053b3147d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be1aa82208a525533eaf88d066fc89da66d75b283a0ab0365e0730a053b3147d/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4cd751abe6f3d1bd40ebaeda185555b87ec00793cc51aea19aa28630130dd0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4cd751abe6f3d1bd40ebaeda185555b87ec00793cc51aea19aa28630130dd0b/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8430f93c69c9ca43f3fe2725bd858512bf0e99e2459f2a05f9088a40e383430/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8430f93c69c9ca43f3fe2725bd858512bf0e99e2459f2a05f9088a40e383430/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc7303134e85a69c59e996f3b5948a6cb5fc57ae411f334fbf7dc659bd3ecb2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc7303134e85a69c59e996f3b5948a6cb5fc57ae411f334fbf7dc659bd3ecb2d/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3f8aa4f6af4a3c34261eaa5558e9617b146e6b1f6cdf7a4e0b124072e917f5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3f8aa4f6af4a3c34261eaa5558e9617b146e6b1f6cdf7a4e0b124072e917f5f/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e22cc8ce73b0d3b02e3bfcf34809f2fce345dae9b1760d39764bd95f4332970a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e22cc8ce73b0d3b02e3bfcf34809f2fce345dae9b1760d39764bd95f4332970a/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec7e1df591d152e0d8408d32e6f8beee904d14190af055ba81d5c90cbcd22d04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec7e1df591d152e0d8408d32e6f8beee904d14190af055ba81d5c90cbcd22d04/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef7cbda66828c3a9685d9e4499fa9ba1f503c8e504781bde67511421036afe0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef7cbda66828c3a9685d9e4499fa9ba1f503c8e504781bde67511421036afe0d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6ea8465bfba56f081cad2ed99944a67a722f8d0e5f316830e04c051b6898d25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6ea8465bfba56f081cad2ed99944a67a722f8d0e5f316830e04c051b6898d25/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~projected/kube-api-access-t5c28:{mountpoint:/var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~projected/kube-api-access-t5c28 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13295b32-a6cc-4918-bfce-9e7819398cb0/volumes/kubernetes.io~projected/kube-api-access-ln9gq:{mountpoint:/var/lib/kubelet/pods/13295b32-a6cc-4918-bfce-9e7819398cb0/volumes/kubernetes.io~projected/kube-api-access-ln9gq major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/167ba867-3860-40d7-9d55-9c81bf49930b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/167ba867-3860-40d7-9d55-9c81bf49930b/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a65a55c-27e9-4e39-a139-53ae907e0bfd/volumes/kubernetes.io~projected/kube-api-access-8ll6l:{mountpoint:/var/lib/kubelet/pods/1a65a55c-27e9-4e39-a139-53ae907e0bfd/volumes/kubernetes.io~projected/kube-api-access-8ll6l major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27d2c28b-a2b2-4293-ad25-a0e69b939b9c/volumes/kubernetes.io~projected/kube-api-access-ghnfj:{mountpoint:/var/lib/kubelet/pods/27d2c28b-a2b2-4293-ad25-a0e69b939b9c/volumes/kubernetes.io~projected/kube-api-access-ghnfj major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29549d27-df3a-48d0-8e7b-85fb883b4481/volumes/kubernetes.io~projected/kube-api-access-d2h7h:{mountpoint:/var/lib/kubelet/pods/29549d27-df3a-48d0-8e7b-85fb883b4481/volumes/kubernetes.io~projected/kube-api-access-d2h7h major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29549d27-df3a-48d0-8e7b-85fb883b4481/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/29549d27-df3a-48d0-8e7b-85fb883b4481/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b407f39-23cc-432f-a5d7-3a377e4762a5/volumes/kubernetes.io~projected/kube-api-access-jnjng:{mountpoint:/var/lib/kubelet/pods/2b407f39-23cc-432f-a5d7-3a377e4762a5/volumes/kubernetes.io~projected/kube-api-access-jnjng major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b407f39-23cc-432f-a5d7-3a377e4762a5/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/2b407f39-23cc-432f-a5d7-3a377e4762a5/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b407f39-23cc-432f-a5d7-3a377e4762a5/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/2b407f39-23cc-432f-a5d7-3a377e4762a5/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38a4b7b8-168f-4525-81be-d43ef28e4c13/volumes/kubernetes.io~projected/kube-api-access-blc54:{mountpoint:/var/lib/kubelet/pods/38a4b7b8-168f-4525-81be-d43ef28e4c13/volumes/kubernetes.io~projected/kube-api-access-blc54 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47ef705c-0b43-46a9-82ba-fe9a8d426ff3/volumes/kubernetes.io~projected/kube-api-access-9j827:{mountpoint:/var/lib/kubelet/pods/47ef705c-0b43-46a9-82ba-fe9a8d426ff3/volumes/kubernetes.io~projected/kube-api-access-9j827 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/486056f5-ed2f-4a7f-b8c8-1130799cf4b9/volumes/kubernetes.io~projected/kube-api-access-fkj95:{mountpoint:/var/lib/kubelet/pods/486056f5-ed2f-4a7f-b8c8-1130799cf4b9/volumes/kubernetes.io~projected/kube-api-access-fkj95 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/551adda7-a285-4f6e-9daf-b5b9f4410b48/volumes/kubernetes.io~projected/kube-api-access-z9pgj:{mountpoint:/var/lib/kubelet/pods/551adda7-a285-4f6e-9daf-b5b9f4410b48/volumes/kubernetes.io~projected/kube-api-access-z9pgj major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/551adda7-a285-4f6e-9daf-b5b9f4410b48/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/551adda7-a285-4f6e-9daf-b5b9f4410b48/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6dc28a5a-54a6-4530-b383-4156404c6f4a/volumes/kubernetes.io~projected/kube-api-access-8mdxv:{mountpoint:/var/lib/kubelet/pods/6dc28a5a-54a6-4530-b383-4156404c6f4a/volumes/kubernetes.io~projected/kube-api-access-8mdxv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fe0d1a0-3adb-4b33-89b6-1f39fb282399/volumes/kubernetes.io~projected/kube-api-access-22rll:{mountpoint:/var/lib/kubelet/pods/6fe0d1a0-3adb-4b33-89b6-1f39fb282399/volumes/kubernetes.io~projected/kube-api-access-22rll major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~projected/kube-api-access-4k9gj:{mountpoint:/var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~projected/kube-api-access-4k9gj major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85b62a96-cc23-4b22-ade6-5066759bbec6/volumes/kubernetes.io~projected/kube-api-access-dd5pv:{mountpoint:/var/lib/kubelet/pods/85b62a96-cc23-4b22-ade6-5066759bbec6/volumes/kubernetes.io~projected/kube-api-access-dd5pv major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92ba2348-2452-4bd2-92ba-a980fdc8a15f/volumes/kubernetes.io~projected/kube-api-access-zxjhv:{mountpoint:/var/lib/kubelet/pods/92ba2348-2452-4bd2-92ba-a980fdc8a15f/volumes/kubernetes.io~projected/kube-api-access-zxjhv major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a97de0f2-2248-44e3-bee7-9817777a261d/volumes/kubernetes.io~projected/kube-api-access-qld6b:{mountpoint:/var/lib/kubelet/pods/a97de0f2-2248-44e3-bee7-9817777a261d/volumes/kubernetes.io~projected/kube-api-access-qld6b major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b26b3373-a3b2-4a0d-bc89-94bd729b849e/volumes/kubernetes.io~projected/kube-api-access-wpbxf:{mountpoint:/var/lib/kubelet/pods/b26b3373-a3b2-4a0d-bc89-94bd729b849e/volumes/kubernetes.io~projected/kube-api-access-wpbxf major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb2c63b2-119c-4f2e-a7e5-664bf9e9b999/volumes/kubernetes.io~projected/kube-api-access-hnrpz:{mountpoint:/var/lib/kubelet/pods/bb2c63b2-119c-4f2e-a7e5-664bf9e9b999/volumes/kubernetes.io~projected/kube-api-access-hnrpz major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3bd5eaa-c913-4709-8632-5abec1047a5e/volumes/kubernetes.io~projected/kube-api-access-f8qp2:{mountpoint:/var/lib/kubelet/pods/c3bd5eaa-c913-4709-8632-5abec1047a5e/volumes/kubernetes.io~projected/kube-api-access-f8qp2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d21994f9-9c90-4711-aca3-0ebb7bbbcebe/volumes/kubernetes.io~projected/kube-api-access-v6pw6:{mountpoint:/var/lib/kubelet/pods/d21994f9-9c90-4711-aca3-0ebb7bbbcebe/volumes/kubernetes.io~projected/kube-api-access-v6pw6 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de504e55-5beb-4f9a-a08a-b4f4e1f999d6/volumes/kubernetes.io~projected/kube-api-access-qfcrv:{mountpoint:/var/lib/kubelet/pods/de504e55-5beb-4f9a-a08a-b4f4e1f999d6/volumes/kubernetes.io~projected/kube-api-access-qfcrv major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de504e55-5beb-4f9a-a08a-b4f4e1f999d6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/de504e55-5beb-4f9a-a08a-b4f4e1f999d6/volumes/kubernetes.io~secret/cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de9d2339-d6be-4332-921d-ec5717b06699/volumes/kubernetes.io~projected/kube-api-access-8cs2d:{mountpoint:/var/lib/kubelet/pods/de9d2339-d6be-4332-921d-ec5717b06699/volumes/kubernetes.io~projected/kube-api-access-8cs2d major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc/volumes/kubernetes.io~projected/kube-api-access-jdmms:{mountpoint:/var/lib/kubelet/pods/e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc/volumes/kubernetes.io~projected/kube-api-access-jdmms major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8/volumes/kubernetes.io~projected/kube-api-access-bskfk:{mountpoint:/var/lib/kubelet/pods/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8/volumes/kubernetes.io~projected/kube-api-access-bskfk major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e718a01a-548e-4499-95dc-7cbbd6ad6a12/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e718a01a-548e-4499-95dc-7cbbd6ad6a12/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7e6e71c-794e-40ba-acef-7fddf496d546/volumes/kubernetes.io~projected/kube-api-access-9hrbq:{mountpoint:/var/lib/kubelet/pods/e7e6e71c-794e-40ba-acef-7fddf496d546/volumes/kubernetes.io~projected/kube-api-access-9hrbq major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e882582b-e97c-4171-8cff-eba17371f519/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e882582b-e97c-4171-8cff-eba17371f519/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e882582b-e97c-4171-8cff-eba17371f519/volumes/kubernetes.io~projected/kube-api-access-pp6n4:{mountpoint:/var/lib/kubelet/pods/e882582b-e97c-4171-8cff-eba17371f519/volumes/kubernetes.io~projected/kube-api-access-pp6n4 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e882582b-e97c-4171-8cff-eba17371f519/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e882582b-e97c-4171-8cff-eba17371f519/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5/volumes/kubernetes.io~projected/kube-api-access-p4dg9:{mountpoint:/var/lib/kubelet/pods/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5/volumes/kubernetes.io~projected/kube-api-access-p4dg9 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1043899-80a3-47a9-9d1c-0421cd68df00/volumes/kubernetes.io~projected/kube-api-access-pn8dq:{mountpoint:/var/lib/kubelet/pods/f1043899-80a3-47a9-9d1c-0421cd68df00/volumes/kubernetes.io~projected/kube-api-access-pn8dq major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2070749-ec46-433b-975e-470c734e5fe5/volumes/kubernetes.io~projected/kube-api-access-h4cdl:{mountpoint:/var/lib/kubelet/pods/f2070749-ec46-433b-975e-470c734e5fe5/volumes/kubernetes.io~projected/kube-api-access-h4cdl major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2070749-ec46-433b-975e-470c734e5fe5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f2070749-ec46-433b-975e-470c734e5fe5/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4572363-4113-4e99-a34a-1b33a4cc08ff/volumes/kubernetes.io~projected/kube-api-access-94n76:{mountpoint:/var/lib/kubelet/pods/f4572363-4113-4e99-a34a-1b33a4cc08ff/volumes/kubernetes.io~projected/kube-api-access-94n76 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d/volumes/kubernetes.io~projected/kube-api-access-7vq4f:{mountpoint:/var/lib/kubelet/pods/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d/volumes/kubernetes.io~projected/kube-api-access-7vq4f major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ea161cea2431a3bd6f3f3bf308f09e30ee4d79a83760358ca192ba5e16c1d1ba/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8e77a1787bca6adbb0098c6d0dd1f4b1e690d9629aba5e634ff290bff6dcbe23/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b0e9d9cc5667bcb13192cf83c08594fd5f7bc6fcc3bb30e1db5cff5337567baa/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/294bf6869a02623d78aa3552a64bf758229bfb4de33d349a79537de4ccc88f09/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/bc69a5c19490d77f940bad8d3a8eae3a01e864a6410aceae9657e3a45c5f7235/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/91aa3d2534914eef7b0696efb8bfec075134e94e19cccf0992943d17158ef917/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0d8af193924d21a0e79e0f88a192f9c690b66466a664ba25e34b25c42a642d20/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/24a00bc821a69d78cc47431c7659ad2137c218c7e0f65476a298a608ed7b6a9a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/dc04feabc7ae517eb1eea73e1a06f8eec25c4bfc24ed9cddc5274e85799456e4/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/5d7991841f448aecac2d6eeb8661ac0c5523fbc1f3715bf90570a05ab8dbd883/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/d95af78f7b87adc0856768c708aa77dd30dc50c3389c034056b86460ed94faca/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/bb865c792e1743f70690f4fb960f39888978d1bd5b12cb80763db77f9ba50032/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/632e93f34005591a4b956371dfa233e3db061d3dc669d9749769bd7110882a1b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/a77f706530e6298b8ec3d4fba8a01854c9961bee90d896011b0b4c0a5fc03ad0/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/285ac6622d17559b821a6c2a44bb9687d8ee986c677aef1d507b26b009b9494d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/fcfa1f96cdecdba590cfc89b87bd49dad68e32e5df2d64655598f357e7897493/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a3ac616e2b2f014c004276108171cecc904543ad398024a5bac91321e83512d9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/50373ed9ca240a9173503efbff7316e5d7dcdb9f52fb88f90f1a6f52acf69502/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7dc392cbf9bb70c13aa8d7b8be87893d5aa2999637ff03f7c292f4e79606552d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/6cbdbd512f70cda03cc1bf1ca492547ac51dc62d1492deebc53ae7d5a6015539/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/284897e2e2921442dbca3bef6f9828701f1767b971203240bf6585a7ccb05f84/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/26020b6ace032f306d652fbdf57d0d9978271b583c87f1f4bc5b08d5c3d41a9a/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/0c8c8951a1270545d854fba74ce26f1d6e65475916608048778831efebbfe000/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/e3a98583b69a15f9cc9c7610575ebea9119fde6ccd0529e55635dce743ec4d17/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/c7523916d2b057347345ce1a242223e39750275947dc9cd45a6218b297568153/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/adffb70e697151a7126dc8efd08b2b078d1420eeaa3904bd7c4f6dbd18ad20d0/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d3f7616fbe04e9f08f96c93b7307a99d3650eda5fc7ffe5e478c80f50b8ef362/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/a4043dbad093839c50b2a2a9edfdb6c1a76b1019f1160ab74a0e7971c41a3117/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/81250f8b4bd2d6782dcedb6e386197a94f4ce28a3f57a904c2523f49b8575187/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/5cac6191d2935f7fa679f10aefc11a44648e5001cb51b5760a25ab411b53ba5f/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/42c4d849b7886b8dd20c79b625c32af2ba823211292190b6d5a7101e7bf3be23/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/f3474e98655b569f436cdca66e47b4d057b80c39693b4c46ec6446b6e6f46150/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/2299f96ed222a5a602e1b523ef66ddcfd1484686923567b30052beffaad0413b/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/d1ea9f9cf2a3f2fe078c4e42d048062067f35f3ec3586796000d6493ab7076b6/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/08c0a88f2478a93992f0adba54d07d64a89b95d93b3541bc0b473174135ce958/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/b9c967f03caf09cd39930fec67f453e06f8732ea3fc72466ad1956b73d2242df/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/2720a20942ff43e460aef9b9f9cde403e4974dd005a299e1695ddf1b5bf09664/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/ee562f4e9dbf8779d4b6a8f3b181fa5b2464a7ccbeeee13aef9a452dcb21850b/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/31ced36ab7d922e3d0be6a879216252cb1c01e8078a9461c1719d17ecda139d9/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/e8c095821dd93706de009e16147774a9fb72c174f32616fcc42166d01aadac15/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/cb1f43bff5635dcd42ca1e247046269f29e608f3f7d2c3d00c9f656981d3f90d/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/1caf479e502e6d0e9241c333dff4d0906d9ad7e365c522b130f8446d78ff0a93/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/a82e8a87caa73f1985e9a179bc2614a5a568162eeb99855f5b90749a5540a1ca/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/c41ae7fdea3851d2b8599c8468d734c1cead391df8c86a683bcaa4467182277d/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/da3a2e176c0f8d533fe4649b0fed5fe95d06ef2c7af1c755389bb749c2a381b2/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/7c63abafd029e92fd88447fe0d6dfbec771b5c05a8c9f21e8c680bd1b82064d7/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/cd054a6fce687fa1a9e9d5c4d39ad4d3ab2e42ce6a5c34667b01dc12cb643f0b/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/ddcd1f707f2ad850572efe366a403f8784872e5e3d90be1cf4b048700e424a7c/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/397d663548cba7ac1c14d691157c16494c38a7de6aac8545cf3e6140d73ecc27/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/eb3ec4ecac3f99cd59c61f410972a45b27ea79432a494193e443e254c5f8847b/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/3b43e7bf2a6790d93cef1e6a6e0d9982de5b083954dca9c3fa3caaf72d234bba/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/25791c63f9bc2b0d4f15f579a0011f21f56c0a4a91a2849fee8fb808c8c67728/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/28db7241303dacdb8e1a79a617ece4e8c94292c60439d396335f8985de34de34/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/86518bb3d9e1192adb2ae7b42ee3dc3d73d8cb871c07a2cef6cb1a766fca1da1/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2bf303b4d0d6e1b0d95a9bca0dc1ba41012ceb36b3fdc6655bd59f5984883a3d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/17e8fac75ede45f8c611f469f68507bf08b9c0cb249a973803352fff9e1f3e14/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/aa8d875d92942136e77140a64044fc6d861348ccedaa2a286af222e9ea3ad3e0/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/1c083aa29e9fa87f8c96bb7bbcc55348761078ed40eec831440e70794ac47805/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/8cb7ca86df45ff07a9a1bddb59a1366cbed48411071941f1a553410191e3d56a/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/80be7ad0441d85dbb5512a9dae2a48637fede0ab2628badd5ff1182f4618e8aa/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/663e48b9b443327ea3d5e0b20ace5d2d36675e9dad0f26ca75d407e7d8235948/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8ec8b2c9754dc4f2cc5c399051123098df52048061b32bde0933c524edb6fb0e/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.086076 5014 manager.go:217] Machine: {Timestamp:2026-10-30 17:56:27.08563636 +0000 UTC m=+0.073976842 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088112 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22dd22e7a4f3bcd61115b1c4bbcb0d SystemUUID:ec22dd22-e7a4-f3bc-d611-15b1c4bbcb0d BootID:14b31835-175b-4e69-b465-31c55c3aeb54 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa4a243c7ba6af3dddabb42fccd05303b2a068cbef89f42cb7d7616043b8ce87/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6eeef5b820286f6ffc3e108c52811f157c97b0effc4a4484ad24bcf1a0d8a2fd/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e882582b-e97c-4171-8cff-eba17371f519/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec7e1df591d152e0d8408d32e6f8beee904d14190af055ba81d5c90cbcd22d04/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/167ba867-3860-40d7-9d55-9c81bf49930b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80ed2bfab374e0140e9c9253103658cb87017bfb519929d673b82c54d75efd4e/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc/volumes/kubernetes.io~projected/kube-api-access-jdmms DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e7e6e71c-794e-40ba-acef-7fddf496d546/volumes/kubernetes.io~projected/kube-api-access-9hrbq DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4572363-4113-4e99-a34a-1b33a4cc08ff/volumes/kubernetes.io~projected/kube-api-access-94n76 DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e882582b-e97c-4171-8cff-eba17371f519/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/38a4b7b8-168f-4525-81be-d43ef28e4c13/volumes/kubernetes.io~projected/kube-api-access-blc54 DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de504e55-5beb-4f9a-a08a-b4f4e1f999d6/volumes/kubernetes.io~projected/kube-api-access-qfcrv DeviceMajor:0 DeviceMinor:226 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e22cc8ce73b0d3b02e3bfcf34809f2fce345dae9b1760d39764bd95f4332970a/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c15a7e453d497f6a249ff157b6b3a54ba12fa006933cca34c9fe902edc35786/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b26b3373-a3b2-4a0d-bc89-94bd729b849e/volumes/kubernetes.io~projected/kube-api-access-wpbxf DeviceMajor:0 DeviceMinor:228 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be1aa82208a525533eaf88d066fc89da66d75b283a0ab0365e0730a053b3147d/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/551adda7-a285-4f6e-9daf-b5b9f4410b48/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b63b939437dbd872c89617da541a3eb01acc58d7392962e27c89a76f21432fd8/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47ef705c-0b43-46a9-82ba-fe9a8d426ff3/volumes/kubernetes.io~projected/kube-api-access-9j827 DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bb2c63b2-119c-4f2e-a7e5-664bf9e9b999/volumes/kubernetes.io~projected/kube-api-access-hnrpz DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~projected/kube-api-access-4k9gj DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/29549d27-df3a-48d0-8e7b-85fb883b4481/volumes/kubernetes.io~projected/kube-api-access-d2h7h DeviceMajor:0 DeviceMinor:229 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5/volumes/kubernetes.io~projected/kube-api-access-p4dg9 DeviceMajor:0 DeviceMinor:361 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b407f39-23cc-432f-a5d7-3a377e4762a5/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~projected/kube-api-access-t5c28 DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e718a01a-548e-4499-95dc-7cbbd6ad6a12/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a65a55c-27e9-4e39-a139-53ae907e0bfd/volumes/kubernetes.io~projected/kube-api-access-8ll6l DeviceMajor:0 DeviceMinor:334 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12e41e5e05545f170de90c819a3692a0a7b2a6bf52372ecc4a7c0e7b7da34845/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2b407f39-23cc-432f-a5d7-3a377e4762a5/volumes/kubernetes.io~projected/kube-api-access-jnjng DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/551adda7-a285-4f6e-9daf-b5b9f4410b48/volumes/kubernetes.io~projected/kube-api-access-z9pgj DeviceMajor:0 DeviceMinor:220 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4cd751abe6f3d1bd40ebaeda185555b87ec00793cc51aea19aa28630130dd0b/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d02664274cfb9bf9de77c00f6954e5b228ca3ba9ff4e19497d60a0c04a14a91/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:210 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85b62a96-cc23-4b22-ade6-5066759bbec6/volumes/kubernetes.io~projected/kube-api-access-dd5pv DeviceMajor:0 DeviceMinor:339 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6dc72b7434eeea4524819dde3b67eb31988432a1bd628c63952b50316e0946bc/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de504e55-5beb-4f9a-a08a-b4f4e1f999d6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7617edbac47192048c6e4eaf4cdfbcf583584596e790eb4f45e2c0f6f8f3a363/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/29549d27-df3a-48d0-8e7b-85fb883b4481/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/27d2c28b-a2b2-4293-ad25-a0e69b939b9c/volumes/kubernetes.io~projected/kube-api-access-ghnfj DeviceMajor:0 DeviceMinor:227 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6dc28a5a-54a6-4530-b383-4156404c6f4a/volumes/kubernetes.io~projected/kube-api-access-8mdxv DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b407f39-23cc-432f-a5d7-3a377e4762a5/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3f8aa4f6af4a3c34261eaa5558e9617b146e6b1f6cdf7a4e0b124072e917f5f/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80f17718bf4c4d5569bbdbc175f558fe131456e7a301666a7a7be10be782ce51/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f1043899-80a3-47a9-9d1c-0421cd68df00/volumes/kubernetes.io~projected/kube-api-access-pn8dq DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/13295b32-a6cc-4918-bfce-9e7819398cb0/volumes/kubernetes.io~projected/kube-api-access-ln9gq DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1f52a68261f62e38dd998161ceb6188a65e071c483afa4a3b43b1eb4f3ed105/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a97de0f2-2248-44e3-bee7-9817777a261d/volumes/kubernetes.io~projected/kube-api-access-qld6b DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d21994f9-9c90-4711-aca3-0ebb7bbbcebe/volumes/kubernetes.io~projected/kube-api-access-v6pw6 DeviceMajor:0 DeviceMinor:342 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d77e2603499f2b6417e3b8ca36449ddfdac483335f5a25ef05a6c2d5fc81337/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8430f93c69c9ca43f3fe2725bd858512bf0e99e2459f2a05f9088a40e383430/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/486056f5-ed2f-4a7f-b8c8-1130799cf4b9/volumes/kubernetes.io~projected/kube-api-access-fkj95 DeviceMajor:0 DeviceMinor:343 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ce825231d61103e72d15ad82becc935dfe0cbea92fe490337dd2ad2ce9c25ac/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef7cbda66828c3a9685d9e4499fa9ba1f503c8e504781bde67511421036afe0d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c3bd5eaa-c913-4709-8632-5abec1047a5e/volumes/kubernetes.io~projected/kube-api-access-f8qp2 DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6fe0d1a0-3adb-4b33-89b6-1f39fb282399/volumes/kubernetes.io~projected/kube-api-access-22rll DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e2650d8ec1d7c969cf0fee983ebe7c48d0543299c11bcf27c6b7fe631b9ad15/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd720384799e29bce3b473e206fdb2afa4f4010ee72a5faf40f105472a4f29ea/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e882582b-e97c-4171-8cff-eba17371f519/volumes/kubernetes.io~projected/kube-api-access-pp6n4 DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2070749-ec46-433b-975e-470c734e5fe5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/de9d2339-d6be-4332-921d-ec5717b06699/volumes/kubernetes.io~projected/kube-api-access-8cs2d DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2070749-ec46-433b-975e-470c734e5fe5/volumes/kubernetes.io~projected/kube-api-access-h4cdl DeviceMajor:0 DeviceMinor:205 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6ea8465bfba56f081cad2ed99944a67a722f8d0e5f316830e04c051b6898d25/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8/volumes/kubernetes.io~projected/kube-api-access-bskfk DeviceMajor:0 DeviceMinor:224 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d/volumes/kubernetes.io~projected/kube-api-access-7vq4f DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e84fbad55f885a2bb894c95b6d4b4280cd7e6dadb02873fe5aa6d4d447e6201/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/92ba2348-2452-4bd2-92ba-a980fdc8a15f/volumes/kubernetes.io~projected/kube-api-access-zxjhv DeviceMajor:0 DeviceMinor:293 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc7303134e85a69c59e996f3b5948a6cb5fc57ae411f334fbf7dc659bd3ecb2d/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bbc517a66512bc6168c017f3267efca9de6c8a80eeae51d7ce1ee3be77455e1/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1bbc517a66512bc MacAddress:2e:dd:1b:f3:c9:37 Speed:10000 Mtu:8901} {Name:1d77e2603499f2b MacAddress:b2:99:96:3a:a1:91 Speed:10000 Mtu:8901} {Name:6eeef5b820286f6 MacAddress:36:ee:8a:81:05:dc Speed:10000 Mtu:8901} {Name:7617edbac471920 MacAddress:32:3f:d1:1b:6a:53 Speed:10000 Mtu:8901} {Name:80f17718bf4c4d5 MacAddress:ce:76:04:60:07:58 Speed:10000 Mtu:8901} {Name:aa4a243c7ba6af3 MacAddress:0e:dd:57:37:10:07 Speed:10000 Mtu:8901} {Name:b63b939437dbd87 MacAddress:d2:0c:1c:f9:14:da Speed:10000 Mtu:8901} {Name:bd720384799e29b MacAddress:22:40:1c:7b:20:d1 Speed:10000 Mtu:8901} {Name:be1aa82208a5255 MacAddress:36:72:ef:34:58:47 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fd:e4:20:fd Speed:0 Mtu:9001} {Name:br-int MacAddress:02:56:84:62:62:66 Speed:0 Mtu:8901} {Name:c4cd751abe6f3d1 MacAddress:ba:3b:6d:a2:98:1b Speed:10000 Mtu:8901} {Name:c8430f93c69c9ca MacAddress:82:45:16:8e:70:55 Speed:10000 Mtu:8901} {Name:cc7303134e85a69 MacAddress:02:0f:a0:b0:99:a4 Speed:10000 Mtu:8901} {Name:d3f8aa4f6af4a3c MacAddress:16:9e:3f:f1:50:ac Speed:10000 Mtu:8901} {Name:e22cc8ce73b0d3b MacAddress:6e:8a:b7:c8:47:1b Speed:10000 Mtu:8901} {Name:ec7e1df591d152e MacAddress:aa:16:93:bf:b7:9b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fd:e4:20:fd Speed:0 Mtu:9001} {Name:f6ea8465bfba56f MacAddress:9a:31:c1:9a:3d:0c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:b2:89:13:40:22:85 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ae:3b:b8:3b:1c:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.086414 5014 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.086463 5014 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.086808 5014 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.086843 5014 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-62.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087029 5014 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087042 5014 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087059 5014 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087076 5014 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087104 5014 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087312 5014 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087456 5014 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087500 5014 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087518 5014 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087533 5014 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087545 5014 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.087556 5014 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.089692 5014 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.089706 5014 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:56:27.090328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.090187 5014 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:56:27.092723 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.092709 5014 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:56:27.092723 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.092723 5014 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:56:27.092819 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.092809 5014 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 17:56:27.092986 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.092976 5014 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:56:27.093014 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093004 5014 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:56:27.093103 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093094 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:56:27.093131 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093109 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:56:27.093131 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093115 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:56:27.093131 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093120 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:56:27.093131 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093128 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:56:27.093240 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093138 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:56:27.093240 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093144 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:56:27.093240 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093149 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:56:27.093240 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093156 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:56:27.093240 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093161 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:56:27.093240 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093168 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:56:27.093240 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093177 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:56:27.093240 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093199 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:56:27.093240 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.093204 5014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:56:27.096429 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.096414 5014 server.go:1263] "Started kubelet" Oct 30 17:56:27.096513 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.096471 5014 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:56:27.101626 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.096693 5014 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:56:27.101626 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.096749 5014 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:56:27.101626 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.097532 5014 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:56:27.101626 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.097741 5014 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:56:27.096963 ip-10-0-133-62 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:56:27.102448 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.102429 5014 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:56:27.102448 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.102447 5014 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:56:27.103198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.102499 5014 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:50:48 +0000 UTC" deadline="2028-04-17 23:44:31.119107321 +0000 UTC" Oct 30 17:56:27.103198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.102528 5014 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12845h48m4.016581897s" Oct 30 17:56:27.103198 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:27.102577 5014 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-62.ec2.internal\" not found" Oct 30 17:56:27.103198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.103073 5014 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:56:27.104161 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.103892 5014 factory.go:55] Registering systemd factory Oct 30 17:56:27.104161 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.103915 5014 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:56:27.104161 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.104146 5014 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:56:27.104366 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.104165 5014 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:56:27.104366 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.104194 5014 factory.go:153] Registering CRI-O factory Oct 30 17:56:27.104366 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.104211 5014 factory.go:223] Registration of the crio container factory successfully Oct 30 17:56:27.104366 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.104265 5014 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:56:27.104366 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.104284 5014 factory.go:103] Registering Raw factory Oct 30 17:56:27.104366 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.104300 5014 manager.go:1199] Started watching for new ooms in manager Oct 30 17:56:27.104699 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.104655 5014 manager.go:319] Starting recovery of all containers Oct 30 17:56:27.107352 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107295 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2070749-ec46-433b-975e-470c734e5fe5" volumeName="kubernetes.io/secret/f2070749-ec46-433b-975e-470c734e5fe5-serving-cert" seLinuxMountContext="" Oct 30 17:56:27.107444 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107359 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dc28a5a-54a6-4530-b383-4156404c6f4a" volumeName="kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 17:56:27.107444 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107377 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fe0d1a0-3adb-4b33-89b6-1f39fb282399" volumeName="kubernetes.io/projected/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-kube-api-access-22rll" seLinuxMountContext="" Oct 30 17:56:27.107444 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107391 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2070749-ec46-433b-975e-470c734e5fe5" volumeName="kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-snapshots" seLinuxMountContext="" Oct 30 17:56:27.107444 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107405 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1136549d-7491-452c-b396-f839668b5a19" volumeName="kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-bound-sa-token" seLinuxMountContext="" Oct 30 17:56:27.107444 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107418 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1136549d-7491-452c-b396-f839668b5a19" volumeName="kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-installation-pull-secrets" seLinuxMountContext="" Oct 30 17:56:27.107444 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107430 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dc28a5a-54a6-4530-b383-4156404c6f4a" volumeName="kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 17:56:27.107444 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107445 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c06084-1bed-46c5-a576-f6129e21e062" volumeName="kubernetes.io/empty-dir/78c06084-1bed-46c5-a576-f6129e21e062-ca-trust-extracted" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107459 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c06084-1bed-46c5-a576-f6129e21e062" volumeName="kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-kube-api-access-4k9gj" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107473 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb2c63b2-119c-4f2e-a7e5-664bf9e9b999" volumeName="kubernetes.io/projected/bb2c63b2-119c-4f2e-a7e5-664bf9e9b999-kube-api-access-hnrpz" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107486 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de504e55-5beb-4f9a-a08a-b4f4e1f999d6" volumeName="kubernetes.io/secret/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-cert" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107498 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de9d2339-d6be-4332-921d-ec5717b06699" volumeName="kubernetes.io/configmap/de9d2339-d6be-4332-921d-ec5717b06699-telemetry-config" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107512 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27d2c28b-a2b2-4293-ad25-a0e69b939b9c" volumeName="kubernetes.io/projected/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-kube-api-access-ghnfj" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107524 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b407f39-23cc-432f-a5d7-3a377e4762a5" volumeName="kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-default-certificate" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107540 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b407f39-23cc-432f-a5d7-3a377e4762a5" volumeName="kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-stats-auth" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107552 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47ef705c-0b43-46a9-82ba-fe9a8d426ff3" volumeName="kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-daemon-config" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107565 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dc28a5a-54a6-4530-b383-4156404c6f4a" volumeName="kubernetes.io/projected/6dc28a5a-54a6-4530-b383-4156404c6f4a-kube-api-access-8mdxv" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107578 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92ba2348-2452-4bd2-92ba-a980fdc8a15f" volumeName="kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107656 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e718a01a-548e-4499-95dc-7cbbd6ad6a12" volumeName="kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107670 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e882582b-e97c-4171-8cff-eba17371f519" volumeName="kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-config" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107692 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1136549d-7491-452c-b396-f839668b5a19" volumeName="kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-trusted-ca" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107704 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b407f39-23cc-432f-a5d7-3a377e4762a5" volumeName="kubernetes.io/projected/2b407f39-23cc-432f-a5d7-3a377e4762a5-kube-api-access-jnjng" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107717 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d21994f9-9c90-4711-aca3-0ebb7bbbcebe" volumeName="kubernetes.io/projected/d21994f9-9c90-4711-aca3-0ebb7bbbcebe-kube-api-access-v6pw6" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107730 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e882582b-e97c-4171-8cff-eba17371f519" volumeName="kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-script-lib" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107742 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5" volumeName="kubernetes.io/secret/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-key" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107754 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4572363-4113-4e99-a34a-1b33a4cc08ff" volumeName="kubernetes.io/projected/f4572363-4113-4e99-a34a-1b33a4cc08ff-kube-api-access-94n76" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107781 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53a361f-64db-41a3-abd1-c04251831048" volumeName="kubernetes.io/configmap/f53a361f-64db-41a3-abd1-c04251831048-nginx-conf" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107794 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd43387-11ba-4288-a0f3-5a16a7d3ce7d" volumeName="kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-tmp" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107812 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1136549d-7491-452c-b396-f839668b5a19" volumeName="kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-registry-certificates" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107823 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="551adda7-a285-4f6e-9daf-b5b9f4410b48" volumeName="kubernetes.io/projected/551adda7-a285-4f6e-9daf-b5b9f4410b48-kube-api-access-z9pgj" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107834 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c06084-1bed-46c5-a576-f6129e21e062" volumeName="kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-bound-sa-token" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107845 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85b62a96-cc23-4b22-ade6-5066759bbec6" volumeName="kubernetes.io/projected/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-api-access-dd5pv" seLinuxMountContext="" Oct 30 17:56:27.107843 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107858 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a51993-67ad-4e39-85fa-0f75e7cbe5b8" volumeName="kubernetes.io/projected/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-kube-api-access-bskfk" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107869 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e882582b-e97c-4171-8cff-eba17371f519" volumeName="kubernetes.io/secret/e882582b-e97c-4171-8cff-eba17371f519-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107882 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5" volumeName="kubernetes.io/projected/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-kube-api-access-p4dg9" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107900 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1043899-80a3-47a9-9d1c-0421cd68df00" volumeName="kubernetes.io/empty-dir/f1043899-80a3-47a9-9d1c-0421cd68df00-tmp-dir" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107914 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a65a55c-27e9-4e39-a139-53ae907e0bfd" volumeName="kubernetes.io/projected/1a65a55c-27e9-4e39-a139-53ae907e0bfd-kube-api-access-8ll6l" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107927 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dc28a5a-54a6-4530-b383-4156404c6f4a" volumeName="kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-binary-copy" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107938 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c06084-1bed-46c5-a576-f6129e21e062" volumeName="kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-registry-certificates" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107950 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1136549d-7491-452c-b396-f839668b5a19" volumeName="kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-image-registry-private-configuration" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107964 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c06084-1bed-46c5-a576-f6129e21e062" volumeName="kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-installation-pull-secrets" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107976 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de9d2339-d6be-4332-921d-ec5717b06699" volumeName="kubernetes.io/projected/de9d2339-d6be-4332-921d-ec5717b06699-kube-api-access-8cs2d" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.107989 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27d2c28b-a2b2-4293-ad25-a0e69b939b9c" volumeName="kubernetes.io/configmap/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-config" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108003 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47ef705c-0b43-46a9-82ba-fe9a8d426ff3" volumeName="kubernetes.io/projected/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-kube-api-access-9j827" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108014 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c06084-1bed-46c5-a576-f6129e21e062" volumeName="kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-image-registry-private-configuration" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108026 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e882582b-e97c-4171-8cff-eba17371f519" volumeName="kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-env-overrides" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108039 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2070749-ec46-433b-975e-470c734e5fe5" volumeName="kubernetes.io/projected/f2070749-ec46-433b-975e-470c734e5fe5-kube-api-access-h4cdl" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108051 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1136549d-7491-452c-b396-f839668b5a19" volumeName="kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-kube-api-access-t5c28" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108065 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c06084-1bed-46c5-a576-f6129e21e062" volumeName="kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-trusted-ca" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108077 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3bd5eaa-c913-4709-8632-5abec1047a5e" volumeName="kubernetes.io/configmap/c3bd5eaa-c913-4709-8632-5abec1047a5e-iptables-alerter-script" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108091 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2070749-ec46-433b-975e-470c734e5fe5" volumeName="kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-tmp" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108103 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4572363-4113-4e99-a34a-1b33a4cc08ff" volumeName="kubernetes.io/configmap/f4572363-4113-4e99-a34a-1b33a4cc08ff-serviceca" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108116 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a65a55c-27e9-4e39-a139-53ae907e0bfd" volumeName="kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-service-ca-bundle" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108128 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a97de0f2-2248-44e3-bee7-9817777a261d" volumeName="kubernetes.io/projected/a97de0f2-2248-44e3-bee7-9817777a261d-kube-api-access-qld6b" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108140 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de504e55-5beb-4f9a-a08a-b4f4e1f999d6" volumeName="kubernetes.io/projected/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-kube-api-access-qfcrv" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108151 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e6e71c-794e-40ba-acef-7fddf496d546" volumeName="kubernetes.io/projected/e7e6e71c-794e-40ba-acef-7fddf496d546-kube-api-access-9hrbq" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108189 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a65a55c-27e9-4e39-a139-53ae907e0bfd" volumeName="kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108202 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29549d27-df3a-48d0-8e7b-85fb883b4481" volumeName="kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-config" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108215 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29549d27-df3a-48d0-8e7b-85fb883b4481" volumeName="kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-trusted-ca" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108227 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="486056f5-ed2f-4a7f-b8c8-1130799cf4b9" volumeName="kubernetes.io/projected/486056f5-ed2f-4a7f-b8c8-1130799cf4b9-kube-api-access-fkj95" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108239 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b26b3373-a3b2-4a0d-bc89-94bd729b849e" volumeName="kubernetes.io/empty-dir/b26b3373-a3b2-4a0d-bc89-94bd729b849e-tmp-dir" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108251 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc" volumeName="kubernetes.io/projected/e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc-kube-api-access-jdmms" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108263 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2070749-ec46-433b-975e-470c734e5fe5" volumeName="kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108276 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="551adda7-a285-4f6e-9daf-b5b9f4410b48" volumeName="kubernetes.io/configmap/551adda7-a285-4f6e-9daf-b5b9f4410b48-config" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108289 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b26b3373-a3b2-4a0d-bc89-94bd729b849e" volumeName="kubernetes.io/configmap/b26b3373-a3b2-4a0d-bc89-94bd729b849e-config-volume" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108300 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b26b3373-a3b2-4a0d-bc89-94bd729b849e" volumeName="kubernetes.io/projected/b26b3373-a3b2-4a0d-bc89-94bd729b849e-kube-api-access-wpbxf" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108313 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e882582b-e97c-4171-8cff-eba17371f519" volumeName="kubernetes.io/projected/e882582b-e97c-4171-8cff-eba17371f519-kube-api-access-pp6n4" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108325 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1043899-80a3-47a9-9d1c-0421cd68df00" volumeName="kubernetes.io/projected/f1043899-80a3-47a9-9d1c-0421cd68df00-kube-api-access-pn8dq" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108338 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2070749-ec46-433b-975e-470c734e5fe5" volumeName="kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-service-ca-bundle" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108351 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd43387-11ba-4288-a0f3-5a16a7d3ce7d" volumeName="kubernetes.io/projected/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-kube-api-access-7vq4f" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108365 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29549d27-df3a-48d0-8e7b-85fb883b4481" volumeName="kubernetes.io/secret/29549d27-df3a-48d0-8e7b-85fb883b4481-serving-cert" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108377 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38a4b7b8-168f-4525-81be-d43ef28e4c13" volumeName="kubernetes.io/projected/38a4b7b8-168f-4525-81be-d43ef28e4c13-kube-api-access-blc54" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108388 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47ef705c-0b43-46a9-82ba-fe9a8d426ff3" volumeName="kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cni-binary-copy" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108409 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="551adda7-a285-4f6e-9daf-b5b9f4410b48" volumeName="kubernetes.io/secret/551adda7-a285-4f6e-9daf-b5b9f4410b48-serving-cert" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108421 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85b62a96-cc23-4b22-ade6-5066759bbec6" volumeName="kubernetes.io/empty-dir/85b62a96-cc23-4b22-ade6-5066759bbec6-data-volume" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108433 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a51993-67ad-4e39-85fa-0f75e7cbe5b8" volumeName="kubernetes.io/configmap/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-config" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108446 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5" volumeName="kubernetes.io/configmap/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-cabundle" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108457 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd43387-11ba-4288-a0f3-5a16a7d3ce7d" volumeName="kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-tuned" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108469 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13295b32-a6cc-4918-bfce-9e7819398cb0" volumeName="kubernetes.io/projected/13295b32-a6cc-4918-bfce-9e7819398cb0-kube-api-access-ln9gq" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108481 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="167ba867-3860-40d7-9d55-9c81bf49930b" volumeName="kubernetes.io/secret/167ba867-3860-40d7-9d55-9c81bf49930b-agent-certs" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108492 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a65a55c-27e9-4e39-a139-53ae907e0bfd" volumeName="kubernetes.io/empty-dir/1a65a55c-27e9-4e39-a139-53ae907e0bfd-archives-path" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108504 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29549d27-df3a-48d0-8e7b-85fb883b4481" volumeName="kubernetes.io/projected/29549d27-df3a-48d0-8e7b-85fb883b4481-kube-api-access-d2h7h" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108517 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1136549d-7491-452c-b396-f839668b5a19" volumeName="kubernetes.io/empty-dir/1136549d-7491-452c-b396-f839668b5a19-ca-trust-extracted" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108530 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="167ba867-3860-40d7-9d55-9c81bf49930b" volumeName="kubernetes.io/configmap/167ba867-3860-40d7-9d55-9c81bf49930b-konnectivity-ca" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108541 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3bd5eaa-c913-4709-8632-5abec1047a5e" volumeName="kubernetes.io/projected/c3bd5eaa-c913-4709-8632-5abec1047a5e-kube-api-access-f8qp2" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108554 5014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a51993-67ad-4e39-85fa-0f75e7cbe5b8" volumeName="kubernetes.io/secret/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-serving-cert" seLinuxMountContext="" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108567 5014 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:56:27.109017 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.108574 5014 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:56:27.132238 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.132184 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 17:56:27.133487 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.133431 5014 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:56:27.134523 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.134498 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 17:56:27.134974 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.134946 5014 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:56:27.134974 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.134972 5014 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:56:27.135111 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.135001 5014 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:56:27.135111 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:27.135042 5014 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:56:27.150203 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.150182 5014 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 17:56:27.161804 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.161784 5014 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:56:27.161938 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.161919 5014 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:56:27.162065 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.162030 5014 generic.go:356] "Generic (PLEG): container finished" podID="f8da8176727de6d4893cff0cfaade9af" containerID="aedf5d7db9427ac1c9b084153689f10cd848a09f2eccd411285bc674c3231751" exitCode=0 Oct 30 17:56:27.163215 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.163193 5014 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:56:27.163461 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.163446 5014 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:56:27.178526 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.178507 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 17:56:27.178620 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.178540 5014 generic.go:356] "Generic (PLEG): container finished" podID="29549d27-df3a-48d0-8e7b-85fb883b4481" containerID="c3757e8548f18241b7a39f1041137562c13252a9639c01546a9db51c789e0c49" exitCode=255 Oct 30 17:56:27.187864 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.187841 5014 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="0be5b1a5e89414aa9942f542f33fed838e0015ecc385d605edb82b3d56993c5e" exitCode=0 Oct 30 17:56:27.187943 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.187864 5014 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="45a1ef7203ada532e26fb6e8c7625b591d99715f6faecdecdf7afadfb7d4e6d2" exitCode=0 Oct 30 17:56:27.187943 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.187874 5014 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="0db10c05b12de4d42967e72bb78b5c297eb5d2e44f76950f71b5086079735ac7" exitCode=0 Oct 30 17:56:27.187943 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.187887 5014 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="4c934ac67c6ce6bb75ca1a0469005bf547f3ba82f6839ad9102d65181f8a5276" exitCode=0 Oct 30 17:56:27.187943 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.187897 5014 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="8d4575654dc7cf557242654e8bb9726c6fa6c81adb46f339c5e13fe3c3fa34d1" exitCode=0 Oct 30 17:56:27.187943 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.187907 5014 generic.go:356] "Generic (PLEG): container finished" podID="6dc28a5a-54a6-4530-b383-4156404c6f4a" containerID="e54e92ac0e79f62391f8e7ff4940f454e9f638b4f824e0ef2f830e2716024dd5" exitCode=0 Oct 30 17:56:27.206168 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.206151 5014 manager.go:324] Recovery completed Oct 30 17:56:27.227201 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227182 5014 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:56:27.227201 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227197 5014 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:56:27.227307 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227216 5014 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:56:27.227347 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227333 5014 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 17:56:27.227377 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227348 5014 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 17:56:27.227377 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227365 5014 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 17:56:27.227377 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227370 5014 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 17:56:27.227377 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227375 5014 policy_none.go:50] "Start" Oct 30 17:56:27.227486 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227383 5014 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:56:27.227486 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227393 5014 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:56:27.227540 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227508 5014 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 17:56:27.227540 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227515 5014 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 17:56:27.227540 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.227520 5014 policy_none.go:44] "Start" Oct 30 17:56:27.232026 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.232002 5014 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:56:27.234651 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.234639 5014 manager.go:341] "Starting Device Plugin manager" Oct 30 17:56:27.235195 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:27.235180 5014 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 17:56:27.235642 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:27.235628 5014 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:56:27.235642 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.235642 5014 server.go:87] "Starting device plugin registration server" Oct 30 17:56:27.235873 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.235862 5014 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:56:27.235921 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.235887 5014 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:56:27.235996 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.235968 5014 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:56:27.236110 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.236072 5014 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:56:27.236218 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.236116 5014 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:56:27.236218 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.236123 5014 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:56:27.236218 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.236160 5014 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:56:27.23610554Z","UUID":"587e4d25-2919-4947-8d62-0b228fbf309c","Handler":null,"Name":"","Endpoint":""} Oct 30 17:56:27.237436 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.237415 5014 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:56:27.237511 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.237443 5014 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:56:27.237560 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:27.237532 5014 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:56:27.336897 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.336881 5014 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:56:27.337968 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.337949 5014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:56:27.338060 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.337979 5014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:56:27.338060 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.337990 5014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-62.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:56:27.338060 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.338037 5014 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.373397 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.373375 5014 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.373482 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.373431 5014 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.436062 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.436001 5014 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal"] Oct 30 17:56:27.436643 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.436629 5014 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.436692 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.436644 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" event={"ID":"f8da8176727de6d4893cff0cfaade9af","Type":"ContainerStarted","Data":"9f7578f2df45ec7e47bcd62f57c351a4d99fbb380b3a5dd2065b4fab177450b5"} Oct 30 17:56:27.436692 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.436683 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" event={"ID":"f8da8176727de6d4893cff0cfaade9af","Type":"ContainerDied","Data":"aedf5d7db9427ac1c9b084153689f10cd848a09f2eccd411285bc674c3231751"} Oct 30 17:56:27.436751 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.436706 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" event={"ID":"f8da8176727de6d4893cff0cfaade9af","Type":"ContainerStarted","Data":"ef7cbda66828c3a9685d9e4499fa9ba1f503c8e504781bde67511421036afe0d"} Oct 30 17:56:27.436751 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.436719 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" event={"ID":"cea7e9c368b2e2c6cdd7c0cacd3b509e","Type":"ContainerStarted","Data":"3106549653c623fc69a8e20c47d26294dfee858cda5bff51cab3b82b83e4cf1d"} Oct 30 17:56:27.436751 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.436730 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" event={"ID":"cea7e9c368b2e2c6cdd7c0cacd3b509e","Type":"ContainerStarted","Data":"8ce825231d61103e72d15ad82becc935dfe0cbea92fe490337dd2ad2ce9c25ac"} Oct 30 17:56:27.436835 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.436810 5014 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.462073 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.462053 5014 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:56:27.462141 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:27.462086 5014 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-62.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.462306 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.462291 5014 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:56:27.462341 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:27.462326 5014 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.552687 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.552664 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cea7e9c368b2e2c6cdd7c0cacd3b509e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-62.ec2.internal\" (UID: \"cea7e9c368b2e2c6cdd7c0cacd3b509e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.552769 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.552694 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.552769 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.552710 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.653216 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.653192 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.653291 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.653221 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.653291 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.653236 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cea7e9c368b2e2c6cdd7c0cacd3b509e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-62.ec2.internal\" (UID: \"cea7e9c368b2e2c6cdd7c0cacd3b509e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.653291 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.653272 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cea7e9c368b2e2c6cdd7c0cacd3b509e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-62.ec2.internal\" (UID: \"cea7e9c368b2e2c6cdd7c0cacd3b509e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.653387 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.653311 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:56:27.653387 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:27.653315 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8da8176727de6d4893cff0cfaade9af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" (UID: \"f8da8176727de6d4893cff0cfaade9af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:56:28.088117 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.088093 5014 apiserver.go:52] "Watching apiserver" Oct 30 17:56:28.138818 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.138792 5014 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:56:28.139475 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.139451 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9hqsk","kube-system/konnectivity-agent-wv5gn","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn","openshift-insights/insights-runtime-extractor-9gjwg","openshift-dns/node-resolver-94ttz","openshift-ingress/router-default-c5cf884d7-2k5bz","openshift-dns/dns-default-867jv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-ddxgl","openshift-cluster-node-tuning-operator/tuned-nmtmw","openshift-service-ca/service-ca-5fffc97bdf-lpb2n","kube-system/global-pull-secret-syncer-2pp59","openshift-network-console/networking-console-plugin-86c478bb88-htdwd","openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f","openshift-image-registry/image-registry-5b849f747d-p4c4m","openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj","openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj","openshift-network-operator/iptables-alerter-t92pz","openshift-console-operator/console-operator-7cd56b8dbc-5mn6l","kube-system/kas-connection-checker-685895c7c9-dh588","openshift-multus/network-metrics-daemon-z4mc4","openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx","kube-system/kas-connection-checker-685895c7c9-4znfd","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98","openshift-ingress-canary/ingress-canary-vj5k4","openshift-insights/insights-operator-7c4986d449-j4bgx","openshift-image-registry/image-registry-698f87866c-k4grf","openshift-image-registry/node-ca-l67qw","openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf","openshift-multus/multus-additional-cni-plugins-bg76c","kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal","openshift-network-diagnostics/network-check-target-lzl5g","openshift-insights/periodic-gathering-fqlng-zdr42","openshift-multus/multus-f5mwt"] Oct 30 17:56:28.140847 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.140828 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.141398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.141379 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.144425 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.144388 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:28.146148 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.146122 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:28.147053 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.147035 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:28.150568 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.150552 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:28.152542 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.152526 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.153171 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.153159 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-867jv" Oct 30 17:56:28.154810 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.154794 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:56:28.154810 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.154813 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 17:56:28.154922 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.154871 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.154922 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.154911 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.155171 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.155044 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.155171 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.155058 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.155948 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.155931 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 17:56:28.156037 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.155988 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 17:56:28.157142 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.157121 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-m9gkv\"" Oct 30 17:56:28.157298 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.157275 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.157416 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.157393 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.157512 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.157125 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 17:56:28.157673 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.157654 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 17:56:28.157955 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.157932 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 17:56:28.158102 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.158079 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 17:56:28.158369 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.158346 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 17:56:28.158466 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.158438 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4qvdt\"" Oct 30 17:56:28.158466 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.158459 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.167571 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.167553 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 17:56:28.167859 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.167844 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 17:56:28.168303 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168286 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vkdz6\"" Oct 30 17:56:28.168387 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168329 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 17:56:28.168387 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168341 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bw9tq\"" Oct 30 17:56:28.168387 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168347 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 17:56:28.168387 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168343 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.168387 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168385 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 17:56:28.168646 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168500 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.168646 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168568 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.168646 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168573 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.168793 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168658 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 17:56:28.168793 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168701 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:56:28.168793 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168707 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.168793 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168741 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.168793 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168770 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.168793 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168772 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 17:56:28.168793 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168781 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.168793 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168791 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wj68l\"" Oct 30 17:56:28.169075 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168800 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:56:28.169075 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168803 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 17:56:28.169075 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168830 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-txfz9\"" Oct 30 17:56:28.169075 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.168879 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v7j5r\"" Oct 30 17:56:28.169075 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.169009 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 17:56:28.169075 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.169041 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dwcjh\"" Oct 30 17:56:28.169425 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.169409 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bkbt2\"" Oct 30 17:56:28.173047 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173024 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 17:56:28.173830 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173808 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.173939 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173837 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:56:28.173939 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173837 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-f7vbx\"" Oct 30 17:56:28.173939 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173862 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.173939 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173869 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:56:28.173939 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173871 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.173939 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173894 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 17:56:28.173939 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173916 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:56:28.173939 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173937 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173946 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173953 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173996 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.173999 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174002 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174029 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174032 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174039 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174055 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174097 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174125 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174131 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174155 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174163 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174189 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174198 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174210 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174233 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174238 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bn4bk\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174248 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174256 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174248 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174248 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-mjzvg\"" Oct 30 17:56:28.174301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174241 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174681 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174686 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174712 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174751 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174751 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174788 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174818 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174826 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174845 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2dl6z\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174845 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z6vc6\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174881 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174887 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174920 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.174931 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gbn2d\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175002 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175017 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-p2k4l\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175024 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dxtgs\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175069 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z9xqn\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175092 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zxlb9\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175158 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8vdz2\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175159 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-26whl\"" Oct 30 17:56:28.175198 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175193 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pc6fd\"" Oct 30 17:56:28.175938 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175324 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tswqq\"" Oct 30 17:56:28.175938 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175360 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:56:28.175938 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175482 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksmnn\"" Oct 30 17:56:28.175938 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.175550 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tzlmk\"" Oct 30 17:56:28.179059 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.179040 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:56:28.179059 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.179041 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:56:28.179214 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.179037 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:56:28.179214 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.179074 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mmrm7\"" Oct 30 17:56:28.179214 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.179075 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7j6rv\"" Oct 30 17:56:28.179893 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.179874 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 17:56:28.180266 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.180247 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 17:56:28.180331 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.180249 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 17:56:28.180331 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.180282 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 17:56:28.180331 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.180304 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pjbxh\"" Oct 30 17:56:28.180450 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.180383 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 17:56:28.192930 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.192915 5014 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:56:28.193054 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.193040 5014 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:56:28.205499 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.205481 5014 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:56:28.215769 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.215753 5014 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:56:28.215836 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.215786 5014 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-62.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-62.ec2.internal" Oct 30 17:56:28.216285 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.216272 5014 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:56:28.216320 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.216304 5014 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-62.ec2.internal" Oct 30 17:56:28.256006 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.255988 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29549d27-df3a-48d0-8e7b-85fb883b4481-serving-cert\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:28.256006 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256014 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2h7h\" (UniqueName: \"kubernetes.io/projected/29549d27-df3a-48d0-8e7b-85fb883b4481-kube-api-access-d2h7h\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:28.256126 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256031 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3bd5eaa-c913-4709-8632-5abec1047a5e-iptables-alerter-script\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:56:28.256126 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256048 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-slash\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.256126 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256064 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-socket-dir-parent\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.256126 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256085 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:28.256126 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256109 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-sys-fs\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.256353 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256151 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.256353 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256234 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-netns\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.256353 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256289 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2070749-ec46-433b-975e-470c734e5fe5-serving-cert\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.256353 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256293 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29549d27-df3a-48d0-8e7b-85fb883b4481-serving-cert\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:28.256353 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256320 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vq4f\" (UniqueName: \"kubernetes.io/projected/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-kube-api-access-7vq4f\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.256353 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256348 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qld6b\" (UniqueName: \"kubernetes.io/projected/a97de0f2-2248-44e3-bee7-9817777a261d-kube-api-access-qld6b\") pod \"kas-connection-checker-685895c7c9-9hqsk\" (UID: \"a97de0f2-2248-44e3-bee7-9817777a261d\") " pod="kube-system/kas-connection-checker-685895c7c9-9hqsk" Oct 30 17:56:28.256675 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256374 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-cert\") pod \"ingress-canary-vj5k4\" (UID: \"de504e55-5beb-4f9a-a08a-b4f4e1f999d6\") " pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:28.256675 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256399 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-key\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:28.256675 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256432 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-service-ca-bundle\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.256675 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256444 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3bd5eaa-c913-4709-8632-5abec1047a5e-iptables-alerter-script\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:56:28.256675 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256461 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-system-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.256675 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256486 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-systemd\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.256675 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256508 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-etc-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.256675 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256584 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cnibin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.256675 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256643 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-service-ca-bundle\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.256675 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256661 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-registry-certificates\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256679 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-cert\") pod \"ingress-canary-vj5k4\" (UID: \"de504e55-5beb-4f9a-a08a-b4f4e1f999d6\") " pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256688 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-cabundle\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256725 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-system-cni-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256747 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a65a55c-27e9-4e39-a139-53ae907e0bfd-archives-path\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256768 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-trusted-ca\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256807 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2070749-ec46-433b-975e-470c734e5fe5-serving-cert\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256827 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a65a55c-27e9-4e39-a139-53ae907e0bfd-archives-path\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256838 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-registry-certificates\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256875 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-var-lib-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256907 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-stats-auth\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256937 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-cabundle\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256943 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjng\" (UniqueName: \"kubernetes.io/projected/2b407f39-23cc-432f-a5d7-3a377e4762a5-kube-api-access-jnjng\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256970 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-signing-key\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.256980 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-trusted-ca\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257018 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdxv\" (UniqueName: \"kubernetes.io/projected/6dc28a5a-54a6-4530-b383-4156404c6f4a-kube-api-access-8mdxv\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257063 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-trusted-ca\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257069 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-kubelet-config\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257066 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-stats-auth\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257106 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-multus-certs\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257134 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-bound-sa-token\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.257160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257159 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cni-binary-copy\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257192 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-etc-kubernetes\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257214 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkj95\" (UniqueName: \"kubernetes.io/projected/486056f5-ed2f-4a7f-b8c8-1130799cf4b9-kube-api-access-fkj95\") pod \"migrator-66df5d496b-qsspf\" (UID: \"486056f5-ed2f-4a7f-b8c8-1130799cf4b9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257231 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257265 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-trusted-ca\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257265 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ll6l\" (UniqueName: \"kubernetes.io/projected/1a65a55c-27e9-4e39-a139-53ae907e0bfd-kube-api-access-8ll6l\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257620 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-kubernetes\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257645 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cni-binary-copy\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257672 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-image-registry-private-configuration\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257707 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-netd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257740 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp6n4\" (UniqueName: \"kubernetes.io/projected/e882582b-e97c-4171-8cff-eba17371f519-kube-api-access-pp6n4\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257767 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257803 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4dg9\" (UniqueName: \"kubernetes.io/projected/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-kube-api-access-p4dg9\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257838 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.257872 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-installation-pull-secrets\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258097 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-installation-pull-secrets\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.258160 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258116 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-image-registry-private-configuration\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.258942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258335 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-kubelet\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.258942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258375 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-snapshots\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.258942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258499 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cdl\" (UniqueName: \"kubernetes.io/projected/f2070749-ec46-433b-975e-470c734e5fe5-kube-api-access-h4cdl\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.258942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258624 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-snapshots\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.258942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258688 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-registry-certificates\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.258942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258830 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4572363-4113-4e99-a34a-1b33a4cc08ff-serviceca\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:56:28.258942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258857 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-registry-certificates\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.258942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258884 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-netns\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.258942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258913 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcrv\" (UniqueName: \"kubernetes.io/projected/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-kube-api-access-qfcrv\") pod \"ingress-canary-vj5k4\" (UID: \"de504e55-5beb-4f9a-a08a-b4f4e1f999d6\") " pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:28.259344 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258959 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-image-registry-private-configuration\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.259344 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.258990 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-bound-sa-token\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.259344 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259049 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.259344 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259097 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n76\" (UniqueName: \"kubernetes.io/projected/f4572363-4113-4e99-a34a-1b33a4cc08ff-kube-api-access-94n76\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:56:28.259344 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259128 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3bd5eaa-c913-4709-8632-5abec1047a5e-host-slash\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:56:28.259344 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259151 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4572363-4113-4e99-a34a-1b33a4cc08ff-serviceca\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:56:28.259344 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259180 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-systemd-units\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.259344 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259235 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-ovn\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.259344 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259279 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1043899-80a3-47a9-9d1c-0421cd68df00-tmp-dir\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:56:28.259823 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259309 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:28.259823 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259403 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-tuned\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.259823 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259435 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b26b3373-a3b2-4a0d-bc89-94bd729b849e-config-volume\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:28.259823 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259463 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.259823 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259575 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1043899-80a3-47a9-9d1c-0421cd68df00-tmp-dir\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:56:28.259823 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259707 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-image-registry-private-configuration\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.260086 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259916 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e718a01a-548e-4499-95dc-7cbbd6ad6a12-original-pull-secret\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:28.260086 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.259920 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b26b3373-a3b2-4a0d-bc89-94bd729b849e-config-volume\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:28.260086 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260003 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.260086 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260026 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-tuned\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.260265 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260146 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.260265 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260182 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-default-certificate\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.260265 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260208 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.260265 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260241 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskfk\" (UniqueName: \"kubernetes.io/projected/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-kube-api-access-bskfk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:28.260454 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260273 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-config\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.260454 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260305 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-trusted-ca-bundle\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:28.260454 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260331 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-tmp\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.260454 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260360 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:28.260454 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260373 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-default-certificate\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.260454 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260389 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/167ba867-3860-40d7-9d55-9c81bf49930b-agent-certs\") pod \"konnectivity-agent-wv5gn\" (UID: \"167ba867-3860-40d7-9d55-9c81bf49930b\") " pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:56:28.260454 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260437 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-bin\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.260777 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260473 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.260777 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260507 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hrbq\" (UniqueName: \"kubernetes.io/projected/e7e6e71c-794e-40ba-acef-7fddf496d546-kube-api-access-9hrbq\") pod \"network-check-source-6f78d59f99-5ks6c\" (UID: \"e7e6e71c-794e-40ba-acef-7fddf496d546\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c" Oct 30 17:56:28.260777 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260535 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4572363-4113-4e99-a34a-1b33a4cc08ff-host\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:56:28.260777 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260620 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.260954 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260828 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-tmp\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.260954 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.260893 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/167ba867-3860-40d7-9d55-9c81bf49930b-agent-certs\") pod \"konnectivity-agent-wv5gn\" (UID: \"167ba867-3860-40d7-9d55-9c81bf49930b\") " pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:56:28.261062 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261043 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-config\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.261113 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261077 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-trusted-ca-bundle\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:28.261113 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261089 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:28.261201 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261110 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-config\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:28.261201 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261153 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5c28\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-kube-api-access-t5c28\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.261201 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261189 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-dbus\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:28.261338 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261221 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j827\" (UniqueName: \"kubernetes.io/projected/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-kube-api-access-9j827\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.261338 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261252 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjhv\" (UniqueName: \"kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv\") pod \"network-check-target-lzl5g\" (UID: \"92ba2348-2452-4bd2-92ba-a980fdc8a15f\") " pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:28.261338 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261282 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-env-overrides\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.261338 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261308 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e882582b-e97c-4171-8cff-eba17371f519-ovn-node-metrics-cert\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.261338 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261325 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-config\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:28.261565 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261340 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-socket-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.261642 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261570 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-env-overrides\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.261909 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261890 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-registration-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.261966 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261933 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:28.261966 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261961 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1043899-80a3-47a9-9d1c-0421cd68df00-hosts-file\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:56:28.262066 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.261985 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e882582b-e97c-4171-8cff-eba17371f519-ovn-node-metrics-cert\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.262066 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262022 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blc54\" (UniqueName: \"kubernetes.io/projected/38a4b7b8-168f-4525-81be-d43ef28e4c13-kube-api-access-blc54\") pod \"kas-connection-checker-685895c7c9-dh588\" (UID: \"38a4b7b8-168f-4525-81be-d43ef28e4c13\") " pod="kube-system/kas-connection-checker-685895c7c9-dh588" Oct 30 17:56:28.262066 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262058 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-modprobe-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.262201 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262089 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnfj\" (UniqueName: \"kubernetes.io/projected/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-kube-api-access-ghnfj\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:28.262201 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262121 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-script-lib\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.262201 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262153 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.262201 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262180 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd5pv\" (UniqueName: \"kubernetes.io/projected/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-api-access-dd5pv\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.262439 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262417 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-k8s-cni-cncf-io\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.262492 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262439 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e882582b-e97c-4171-8cff-eba17371f519-ovnkube-script-lib\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.262492 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262469 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-lib-modules\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.262932 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262687 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/de9d2339-d6be-4332-921d-ec5717b06699-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:28.262932 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262732 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c06084-1bed-46c5-a576-f6129e21e062-ca-trust-extracted\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.262932 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262791 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6pw6\" (UniqueName: \"kubernetes.io/projected/d21994f9-9c90-4711-aca3-0ebb7bbbcebe-kube-api-access-v6pw6\") pod \"migrator-66df5d496b-kz75f\" (UID: \"d21994f9-9c90-4711-aca3-0ebb7bbbcebe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f" Oct 30 17:56:28.262932 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262823 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26b3373-a3b2-4a0d-bc89-94bd729b849e-tmp-dir\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:28.263165 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.262908 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.263165 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263026 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c06084-1bed-46c5-a576-f6129e21e062-ca-trust-extracted\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.263261 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263251 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/de9d2339-d6be-4332-921d-ec5717b06699-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:28.263331 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263271 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26b3373-a3b2-4a0d-bc89-94bd729b849e-tmp-dir\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:28.263696 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263678 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b62a96-cc23-4b22-ade6-5066759bbec6-data-volume\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.263751 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263720 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-service-ca-bundle\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:28.263751 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263733 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b62a96-cc23-4b22-ade6-5066759bbec6-data-volume\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.263751 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263746 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:28.263895 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263772 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-log-socket\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.263895 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263797 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/551adda7-a285-4f6e-9daf-b5b9f4410b48-serving-cert\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:28.263895 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263821 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-hostroot\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.263895 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263844 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-daemon-config\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.263895 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263867 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-run\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.263895 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263891 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-tmp\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.263895 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263891 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-service-ca-bundle\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263929 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8qp2\" (UniqueName: \"kubernetes.io/projected/c3bd5eaa-c913-4709-8632-5abec1047a5e-kube-api-access-f8qp2\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263959 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263842 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2070749-ec46-433b-975e-470c734e5fe5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263969 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2070749-ec46-433b-975e-470c734e5fe5-tmp\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263982 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263989 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-kubelet\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.263980 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/551adda7-a285-4f6e-9daf-b5b9f4410b48-serving-cert\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264044 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264068 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264130 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cs2d\" (UniqueName: \"kubernetes.io/projected/de9d2339-d6be-4332-921d-ec5717b06699-kube-api-access-8cs2d\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264156 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dc28a5a-54a6-4530-b383-4156404c6f4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.264166 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264164 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:28.264741 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264222 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-daemon-config\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.264741 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264255 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:28.264741 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264288 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rll\" (UniqueName: \"kubernetes.io/projected/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-kube-api-access-22rll\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:28.264741 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264314 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.264741 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264368 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-multus\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.264741 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264425 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-sys\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.264741 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264627 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:28.265078 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264900 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-host\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.265078 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264941 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k9gj\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-kube-api-access-4k9gj\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.265078 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264970 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/167ba867-3860-40d7-9d55-9c81bf49930b-konnectivity-ca\") pod \"konnectivity-agent-wv5gn\" (UID: \"167ba867-3860-40d7-9d55-9c81bf49930b\") " pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:56:28.265078 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.264986 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-systemd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.265078 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.265004 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrpz\" (UniqueName: \"kubernetes.io/projected/bb2c63b2-119c-4f2e-a7e5-664bf9e9b999-kube-api-access-hnrpz\") pod \"volume-data-source-validator-58c9f5f46d-ggqkn\" (UID: \"bb2c63b2-119c-4f2e-a7e5-664bf9e9b999\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn" Oct 30 17:56:28.265078 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.265028 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln9gq\" (UniqueName: \"kubernetes.io/projected/13295b32-a6cc-4918-bfce-9e7819398cb0-kube-api-access-ln9gq\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.265078 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.265062 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.265398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.265127 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f53a361f-64db-41a3-abd1-c04251831048-nginx-conf\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:28.265398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.265189 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/167ba867-3860-40d7-9d55-9c81bf49930b-konnectivity-ca\") pod \"konnectivity-agent-wv5gn\" (UID: \"167ba867-3860-40d7-9d55-9c81bf49930b\") " pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:56:28.265398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.265205 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-cnibin\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.265398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.265241 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-conf-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.265398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.265273 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbxf\" (UniqueName: \"kubernetes.io/projected/b26b3373-a3b2-4a0d-bc89-94bd729b849e-kube-api-access-wpbxf\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:28.265398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.265299 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-node-log\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.265398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.265323 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-conf\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.265398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.265362 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f53a361f-64db-41a3-abd1-c04251831048-nginx-conf\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:28.266671 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266641 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-device-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.266764 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266684 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.266764 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266714 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmms\" (UniqueName: \"kubernetes.io/projected/e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc-kube-api-access-jdmms\") pod \"kas-connection-checker-685895c7c9-4znfd\" (UID: \"e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc\") " pod="kube-system/kas-connection-checker-685895c7c9-4znfd" Oct 30 17:56:28.266764 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266737 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-var-lib-kubelet\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.266764 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266762 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-trusted-ca\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:28.266942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266786 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:28.266942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266807 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-etc-selinux\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.266942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266832 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-installation-pull-secrets\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.266942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266856 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b62a96-cc23-4b22-ade6-5066759bbec6-crio-socket\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.266942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266878 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-bin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.266942 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266916 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:28.267211 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.266984 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/551adda7-a285-4f6e-9daf-b5b9f4410b48-config\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:28.267211 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.267010 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1136549d-7491-452c-b396-f839668b5a19-ca-trust-extracted\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.267211 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.267036 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8dq\" (UniqueName: \"kubernetes.io/projected/f1043899-80a3-47a9-9d1c-0421cd68df00-kube-api-access-pn8dq\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:56:28.267211 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.267069 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysconfig\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.267211 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.267096 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9pgj\" (UniqueName: \"kubernetes.io/projected/551adda7-a285-4f6e-9daf-b5b9f4410b48-kube-api-access-z9pgj\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:28.267428 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.267254 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29549d27-df3a-48d0-8e7b-85fb883b4481-trusted-ca\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:28.267428 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.267331 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/551adda7-a285-4f6e-9daf-b5b9f4410b48-config\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:28.267428 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.267396 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1136549d-7491-452c-b396-f839668b5a19-ca-trust-extracted\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.267641 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.267620 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-installation-pull-secrets\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.281741 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.281723 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qld6b\" (UniqueName: \"kubernetes.io/projected/a97de0f2-2248-44e3-bee7-9817777a261d-kube-api-access-qld6b\") pod \"kas-connection-checker-685895c7c9-9hqsk\" (UID: \"a97de0f2-2248-44e3-bee7-9817777a261d\") " pod="kube-system/kas-connection-checker-685895c7c9-9hqsk" Oct 30 17:56:28.281819 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.281770 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdxv\" (UniqueName: \"kubernetes.io/projected/6dc28a5a-54a6-4530-b383-4156404c6f4a-kube-api-access-8mdxv\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.285346 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.285327 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp6n4\" (UniqueName: \"kubernetes.io/projected/e882582b-e97c-4171-8cff-eba17371f519-kube-api-access-pp6n4\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.285451 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.285436 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjng\" (UniqueName: \"kubernetes.io/projected/2b407f39-23cc-432f-a5d7-3a377e4762a5-kube-api-access-jnjng\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.288439 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.288422 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-bound-sa-token\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.288485 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.288476 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vq4f\" (UniqueName: \"kubernetes.io/projected/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-kube-api-access-7vq4f\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.288553 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.288538 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2h7h\" (UniqueName: \"kubernetes.io/projected/29549d27-df3a-48d0-8e7b-85fb883b4481-kube-api-access-d2h7h\") pod \"console-operator-7cd56b8dbc-5mn6l\" (UID: \"29549d27-df3a-48d0-8e7b-85fb883b4481\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:28.297242 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.297216 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-bound-sa-token\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.297316 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.297300 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkj95\" (UniqueName: \"kubernetes.io/projected/486056f5-ed2f-4a7f-b8c8-1130799cf4b9-kube-api-access-fkj95\") pod \"migrator-66df5d496b-qsspf\" (UID: \"486056f5-ed2f-4a7f-b8c8-1130799cf4b9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qsspf" Oct 30 17:56:28.297477 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.297461 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ll6l\" (UniqueName: \"kubernetes.io/projected/1a65a55c-27e9-4e39-a139-53ae907e0bfd-kube-api-access-8ll6l\") pod \"periodic-gathering-fqlng-zdr42\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:56:28.297712 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.297690 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4cdl\" (UniqueName: \"kubernetes.io/projected/f2070749-ec46-433b-975e-470c734e5fe5-kube-api-access-h4cdl\") pod \"insights-operator-7c4986d449-j4bgx\" (UID: \"f2070749-ec46-433b-975e-470c734e5fe5\") " pod="openshift-insights/insights-operator-7c4986d449-j4bgx" Oct 30 17:56:28.297796 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.297782 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94n76\" (UniqueName: \"kubernetes.io/projected/f4572363-4113-4e99-a34a-1b33a4cc08ff-kube-api-access-94n76\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:56:28.298177 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.298158 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4dg9\" (UniqueName: \"kubernetes.io/projected/f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5-kube-api-access-p4dg9\") pod \"service-ca-5fffc97bdf-lpb2n\" (UID: \"f0bcd372-b9ea-4a6c-b9ce-b309d04b3ed5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lpb2n" Oct 30 17:56:28.298243 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.298199 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcrv\" (UniqueName: \"kubernetes.io/projected/de504e55-5beb-4f9a-a08a-b4f4e1f999d6-kube-api-access-qfcrv\") pod \"ingress-canary-vj5k4\" (UID: \"de504e55-5beb-4f9a-a08a-b4f4e1f999d6\") " pod="openshift-ingress-canary/ingress-canary-vj5k4" Oct 30 17:56:28.299743 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.299723 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blc54\" (UniqueName: \"kubernetes.io/projected/38a4b7b8-168f-4525-81be-d43ef28e4c13-kube-api-access-blc54\") pod \"kas-connection-checker-685895c7c9-dh588\" (UID: \"38a4b7b8-168f-4525-81be-d43ef28e4c13\") " pod="kube-system/kas-connection-checker-685895c7c9-dh588" Oct 30 17:56:28.299997 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.299978 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hrbq\" (UniqueName: \"kubernetes.io/projected/e7e6e71c-794e-40ba-acef-7fddf496d546-kube-api-access-9hrbq\") pod \"network-check-source-6f78d59f99-5ks6c\" (UID: \"e7e6e71c-794e-40ba-acef-7fddf496d546\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5ks6c" Oct 30 17:56:28.302854 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.302823 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskfk\" (UniqueName: \"kubernetes.io/projected/e3a51993-67ad-4e39-85fa-0f75e7cbe5b8-kube-api-access-bskfk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj\" (UID: \"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" Oct 30 17:56:28.302953 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.302884 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd5pv\" (UniqueName: \"kubernetes.io/projected/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-api-access-dd5pv\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.302953 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.302898 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6pw6\" (UniqueName: \"kubernetes.io/projected/d21994f9-9c90-4711-aca3-0ebb7bbbcebe-kube-api-access-v6pw6\") pod \"migrator-66df5d496b-kz75f\" (UID: \"d21994f9-9c90-4711-aca3-0ebb7bbbcebe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kz75f" Oct 30 17:56:28.303026 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.302966 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln9gq\" (UniqueName: \"kubernetes.io/projected/13295b32-a6cc-4918-bfce-9e7819398cb0-kube-api-access-ln9gq\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.303026 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.303004 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k9gj\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-kube-api-access-4k9gj\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.303102 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.303067 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjhv\" (UniqueName: \"kubernetes.io/projected/92ba2348-2452-4bd2-92ba-a980fdc8a15f-kube-api-access-zxjhv\") pod \"network-check-target-lzl5g\" (UID: \"92ba2348-2452-4bd2-92ba-a980fdc8a15f\") " pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:28.303152 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.303139 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cs2d\" (UniqueName: \"kubernetes.io/projected/de9d2339-d6be-4332-921d-ec5717b06699-kube-api-access-8cs2d\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:28.303528 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.303512 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbxf\" (UniqueName: \"kubernetes.io/projected/b26b3373-a3b2-4a0d-bc89-94bd729b849e-kube-api-access-wpbxf\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:28.305251 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.305234 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9pgj\" (UniqueName: \"kubernetes.io/projected/551adda7-a285-4f6e-9daf-b5b9f4410b48-kube-api-access-z9pgj\") pod \"service-ca-operator-544d749c8c-7xzrx\" (UID: \"551adda7-a285-4f6e-9daf-b5b9f4410b48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" Oct 30 17:56:28.305316 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.305261 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmms\" (UniqueName: \"kubernetes.io/projected/e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc-kube-api-access-jdmms\") pod \"kas-connection-checker-685895c7c9-4znfd\" (UID: \"e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc\") " pod="kube-system/kas-connection-checker-685895c7c9-4znfd" Oct 30 17:56:28.307751 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.307732 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j827\" (UniqueName: \"kubernetes.io/projected/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-kube-api-access-9j827\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.307879 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.307859 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rll\" (UniqueName: \"kubernetes.io/projected/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-kube-api-access-22rll\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:28.307935 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.307867 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8dq\" (UniqueName: \"kubernetes.io/projected/f1043899-80a3-47a9-9d1c-0421cd68df00-kube-api-access-pn8dq\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:56:28.307935 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.307893 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrpz\" (UniqueName: \"kubernetes.io/projected/bb2c63b2-119c-4f2e-a7e5-664bf9e9b999-kube-api-access-hnrpz\") pod \"volume-data-source-validator-58c9f5f46d-ggqkn\" (UID: \"bb2c63b2-119c-4f2e-a7e5-664bf9e9b999\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggqkn" Oct 30 17:56:28.308057 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.308043 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnfj\" (UniqueName: \"kubernetes.io/projected/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-kube-api-access-ghnfj\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:28.309336 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.309291 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8qp2\" (UniqueName: \"kubernetes.io/projected/c3bd5eaa-c913-4709-8632-5abec1047a5e-kube-api-access-f8qp2\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:56:28.312889 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.312871 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5c28\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-kube-api-access-t5c28\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.367570 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367518 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-var-lib-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.367570 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367543 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-kubelet-config\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:28.367708 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367568 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-multus-certs\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.367708 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367613 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-etc-kubernetes\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.367708 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367638 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.367708 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367630 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-var-lib-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.367708 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367656 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-kubelet-config\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:28.367708 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367665 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-kubernetes\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.367708 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367656 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-multus-certs\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.367708 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367706 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367696 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-etc-kubernetes\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367721 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-netd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367708 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-kubernetes\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367689 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-netd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367752 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367767 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367781 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-kubelet\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367806 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-kubelet\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367815 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-netns\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367850 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3bd5eaa-c913-4709-8632-5abec1047a5e-host-slash\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.367881 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:28.867862016 +0000 UTC m=+1.856202498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : configmap references non-existent config key: service-ca.crt Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367895 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3bd5eaa-c913-4709-8632-5abec1047a5e-host-slash\") pod \"iptables-alerter-t92pz\" (UID: \"c3bd5eaa-c913-4709-8632-5abec1047a5e\") " pod="openshift-network-operator/iptables-alerter-t92pz" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367901 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-systemd-units\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367918 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367926 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-ovn\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367946 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-netns\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.367957 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367952 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-systemd-units\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.367975 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368009 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-ovn\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368047 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368074 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368105 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-bin\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368125 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368148 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4572363-4113-4e99-a34a-1b33a4cc08ff-host\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368163 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368170 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-dbus\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368198 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-socket-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368185 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4572363-4113-4e99-a34a-1b33a4cc08ff-host\") pod \"node-ca-l67qw\" (UID: \"f4572363-4113-4e99-a34a-1b33a4cc08ff\") " pod="openshift-image-registry/node-ca-l67qw" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368214 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e718a01a-548e-4499-95dc-7cbbd6ad6a12-dbus\") pod \"global-pull-secret-syncer-2pp59\" (UID: \"e718a01a-548e-4499-95dc-7cbbd6ad6a12\") " pod="kube-system/global-pull-secret-syncer-2pp59" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368221 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-registration-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368171 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-cni-bin\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368261 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-socket-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368266 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-registration-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368267 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368315 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1043899-80a3-47a9-9d1c-0421cd68df00-hosts-file\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.368331 5014 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368341 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-modprobe-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368367 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.368374 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert podName:f53a361f-64db-41a3-abd1-c04251831048 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:28.868361887 +0000 UTC m=+1.856702374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htdwd" (UID: "f53a361f-64db-41a3-abd1-c04251831048") : secret "networking-console-plugin-cert" not found Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368381 5014 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368437 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-k8s-cni-cncf-io\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368406 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-k8s-cni-cncf-io\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368470 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-modprobe-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.368473 5014 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368478 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1043899-80a3-47a9-9d1c-0421cd68df00-hosts-file\") pod \"node-resolver-94ttz\" (UID: \"f1043899-80a3-47a9-9d1c-0421cd68df00\") " pod="openshift-dns/node-resolver-94ttz" Oct 30 17:56:28.368489 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368473 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-lib-modules\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.368514 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:28.868504575 +0000 UTC m=+1.856845075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368552 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-lib-modules\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368555 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-log-socket\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368588 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-hostroot\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368614 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-log-socket\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368625 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-run\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368647 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-kubelet\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368658 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-hostroot\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368662 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368705 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368723 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-run\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368734 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368738 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-kubelet\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368761 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368786 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-multus\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368808 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-sys\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.368814 5014 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368830 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-host\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.368857 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls podName:27d2c28b-a2b2-4293-ad25-a0e69b939b9c nodeName:}" failed. No retries permitted until 2026-10-30 17:56:28.868842385 +0000 UTC m=+1.857182851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgr98" (UID: "27d2c28b-a2b2-4293-ad25-a0e69b939b9c") : secret "samples-operator-tls" not found Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368878 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-host\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368884 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-systemd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368762 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368927 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-run-systemd\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368928 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368926 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-sys\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368954 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-multus\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368965 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-cnibin\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368985 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-conf-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.368997 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-cnibin\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369019 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369034 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-node-log\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369026 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-d\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369001 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-node-log\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369086 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-conf\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369027 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-conf-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369111 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-device-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369137 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369162 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-var-lib-kubelet\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369188 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysctl-conf\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369190 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369218 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-device-dir\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369225 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-etc-selinux\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369237 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-var-lib-kubelet\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369249 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b62a96-cc23-4b22-ade6-5066759bbec6-crio-socket\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.369259 5014 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369283 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-etc-selinux\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369482 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b62a96-cc23-4b22-ade6-5066759bbec6-crio-socket\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.369506 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls podName:b26b3373-a3b2-4a0d-bc89-94bd729b849e nodeName:}" failed. No retries permitted until 2026-10-30 17:56:28.869489374 +0000 UTC m=+1.857829839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls") pod "dns-default-867jv" (UID: "b26b3373-a3b2-4a0d-bc89-94bd729b849e") : secret "dns-default-metrics-tls" not found Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369528 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-bin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369548 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-var-lib-cni-bin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369561 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369609 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysconfig\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369670 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-sysconfig\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369697 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-slash\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.369700 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369721 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-socket-dir-parent\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.369705 5014 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369785 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-multus-socket-dir-parent\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369745 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369832 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-sys-fs\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369734 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-host-slash\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369856 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.369807 5014 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369880 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-netns\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.369910 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls podName:de9d2339-d6be-4332-921d-ec5717b06699 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:28.869895031 +0000 UTC m=+1.858235507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wb2x4" (UID: "de9d2339-d6be-4332-921d-ec5717b06699") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369919 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13295b32-a6cc-4918-bfce-9e7819398cb0-sys-fs\") pod \"aws-ebs-csi-driver-node-tvxtj\" (UID: \"13295b32-a6cc-4918-bfce-9e7819398cb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvxtj" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.369931 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs podName:6fe0d1a0-3adb-4b33-89b6-1f39fb282399 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:28.86992085 +0000 UTC m=+1.858261327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs") pod "network-metrics-daemon-z4mc4" (UID: "6fe0d1a0-3adb-4b33-89b6-1f39fb282399") : secret "metrics-daemon-secret" not found Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369956 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-system-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369961 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-host-run-netns\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.369980 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-systemd\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.370006 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-etc-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.370030 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cnibin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.370031 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-system-cni-dir\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.370057 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-system-cni-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.370078 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd43387-11ba-4288-a0f3-5a16a7d3ce7d-etc-systemd\") pod \"tuned-nmtmw\" (UID: \"ffd43387-11ba-4288-a0f3-5a16a7d3ce7d\") " pod="openshift-cluster-node-tuning-operator/tuned-nmtmw" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.370115 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e882582b-e97c-4171-8cff-eba17371f519-etc-openvswitch\") pod \"ovnkube-node-ddxgl\" (UID: \"e882582b-e97c-4171-8cff-eba17371f519\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.370128 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc28a5a-54a6-4530-b383-4156404c6f4a-system-cni-dir\") pod \"multus-additional-cni-plugins-bg76c\" (UID: \"6dc28a5a-54a6-4530-b383-4156404c6f4a\") " pod="openshift-multus/multus-additional-cni-plugins-bg76c" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.370160 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ef705c-0b43-46a9-82ba-fe9a8d426ff3-cnibin\") pod \"multus-f5mwt\" (UID: \"47ef705c-0b43-46a9-82ba-fe9a8d426ff3\") " pod="openshift-multus/multus-f5mwt" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.370385 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b62a96-cc23-4b22-ade6-5066759bbec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.371089 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b407f39-23cc-432f-a5d7-3a377e4762a5-metrics-certs\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.371398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.371244 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\") pod \"image-registry-698f87866c-k4grf\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.372137 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.371649 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\") pod \"image-registry-5b849f747d-p4c4m\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.441303 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.441274 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:28.445856 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.445839 5014 scope.go:122] "RemoveContainer" containerID="c3757e8548f18241b7a39f1041137562c13252a9639c01546a9db51c789e0c49" Oct 30 17:56:28.452937 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.452922 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:28.772458 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.772380 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b849f747d-p4c4m"] Oct 30 17:56:28.776486 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:28.776459 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1136549d_7491_452c_b396_f839668b5a19.slice/crio-6a895777390d7c2066ff707bac21dff7aa16c1ecfb285fdf0c5daca6da9907e2 WatchSource:0}: Error finding container 6a895777390d7c2066ff707bac21dff7aa16c1ecfb285fdf0c5daca6da9907e2: Status 404 returned error can't find the container with id 6a895777390d7c2066ff707bac21dff7aa16c1ecfb285fdf0c5daca6da9907e2 Oct 30 17:56:28.809023 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.809003 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-698f87866c-k4grf"] Oct 30 17:56:28.814606 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:28.814565 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c06084_1bed_46c5_a576_f6129e21e062.slice/crio-c0be5292269a0a5c60fdcc459fe9621b41198e171416cea79f5fae32e7c1647e WatchSource:0}: Error finding container c0be5292269a0a5c60fdcc459fe9621b41198e171416cea79f5fae32e7c1647e: Status 404 returned error can't find the container with id c0be5292269a0a5c60fdcc459fe9621b41198e171416cea79f5fae32e7c1647e Oct 30 17:56:28.874164 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.874134 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:28.874265 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.874208 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:28.874265 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.874241 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:28.874387 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874287 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:29.874270116 +0000 UTC m=+2.862610585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : configmap references non-existent config key: service-ca.crt Oct 30 17:56:28.874387 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.874314 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:28.874387 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874339 5014 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:56:28.874387 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.874349 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:28.874387 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.874371 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:28.874387 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874344 5014 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:56:28.874731 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874395 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert podName:f53a361f-64db-41a3-abd1-c04251831048 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:29.874378259 +0000 UTC m=+2.862718725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htdwd" (UID: "f53a361f-64db-41a3-abd1-c04251831048") : secret "networking-console-plugin-cert" not found Oct 30 17:56:28.874731 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874404 5014 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:56:28.874731 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874419 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:29.874408476 +0000 UTC m=+2.862748945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:56:28.874731 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874439 5014 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:56:28.874731 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874446 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls podName:b26b3373-a3b2-4a0d-bc89-94bd729b849e nodeName:}" failed. No retries permitted until 2026-10-30 17:56:29.874432453 +0000 UTC m=+2.862772923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls") pod "dns-default-867jv" (UID: "b26b3373-a3b2-4a0d-bc89-94bd729b849e") : secret "dns-default-metrics-tls" not found Oct 30 17:56:28.874731 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:28.874468 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:28.874731 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874489 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls podName:27d2c28b-a2b2-4293-ad25-a0e69b939b9c nodeName:}" failed. No retries permitted until 2026-10-30 17:56:29.874477858 +0000 UTC m=+2.862818323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgr98" (UID: "27d2c28b-a2b2-4293-ad25-a0e69b939b9c") : secret "samples-operator-tls" not found Oct 30 17:56:28.874731 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874513 5014 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:28.874731 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874524 5014 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:56:28.874731 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874539 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls podName:de9d2339-d6be-4332-921d-ec5717b06699 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:29.874531185 +0000 UTC m=+2.862871653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wb2x4" (UID: "de9d2339-d6be-4332-921d-ec5717b06699") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:28.874731 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:28.874557 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs podName:6fe0d1a0-3adb-4b33-89b6-1f39fb282399 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:29.874546923 +0000 UTC m=+2.862887404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs") pod "network-metrics-daemon-z4mc4" (UID: "6fe0d1a0-3adb-4b33-89b6-1f39fb282399") : secret "metrics-daemon-secret" not found Oct 30 17:56:29.198903 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.198875 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 17:56:29.199298 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.198966 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" event={"ID":"29549d27-df3a-48d0-8e7b-85fb883b4481","Type":"ContainerStarted","Data":"45c35670243915a3071eeb0790261a3683ef0a060f24465bc9af87a44cbab772"} Oct 30 17:56:29.200527 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.200493 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" event={"ID":"1136549d-7491-452c-b396-f839668b5a19","Type":"ContainerStarted","Data":"3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e"} Oct 30 17:56:29.200662 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.200541 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" event={"ID":"1136549d-7491-452c-b396-f839668b5a19","Type":"ContainerStarted","Data":"6a895777390d7c2066ff707bac21dff7aa16c1ecfb285fdf0c5daca6da9907e2"} Oct 30 17:56:29.203632 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.202799 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698f87866c-k4grf" event={"ID":"78c06084-1bed-46c5-a576-f6129e21e062","Type":"ContainerStarted","Data":"05cb2273fb6cae9aed02024166258d2fa52277da54b5199cb2b56875c3ba74e0"} Oct 30 17:56:29.203632 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.202843 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698f87866c-k4grf" event={"ID":"78c06084-1bed-46c5-a576-f6129e21e062","Type":"ContainerStarted","Data":"c0be5292269a0a5c60fdcc459fe9621b41198e171416cea79f5fae32e7c1647e"} Oct 30 17:56:29.790979 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.790952 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:29.884404 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.884374 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:29.884551 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.884452 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:29.884551 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.884495 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:29.884551 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:29.884541 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.884521659 +0000 UTC m=+4.872862149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : configmap references non-existent config key: service-ca.crt Oct 30 17:56:29.884713 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:29.884573 5014 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:56:29.884950 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:29.884930 5014 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:56:29.885030 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:29.885009 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.88499089 +0000 UTC m=+4.873331369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:56:29.885030 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:29.884936 5014 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:56:29.885148 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:29.885102 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls podName:27d2c28b-a2b2-4293-ad25-a0e69b939b9c nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.885084574 +0000 UTC m=+4.873425047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgr98" (UID: "27d2c28b-a2b2-4293-ad25-a0e69b939b9c") : secret "samples-operator-tls" not found Oct 30 17:56:29.885205 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.884588 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:29.885268 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.885214 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:29.885268 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:29.885229 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert podName:f53a361f-64db-41a3-abd1-c04251831048 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.885211496 +0000 UTC m=+4.873551979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htdwd" (UID: "f53a361f-64db-41a3-abd1-c04251831048") : secret "networking-console-plugin-cert" not found Oct 30 17:56:29.885365 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.885273 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:29.885412 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:29.885385 5014 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:29.885457 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.885405 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:29.885457 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:29.885447 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls podName:de9d2339-d6be-4332-921d-ec5717b06699 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.885433438 +0000 UTC m=+4.873773918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wb2x4" (UID: "de9d2339-d6be-4332-921d-ec5717b06699") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:29.885555 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:29.885493 5014 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:56:29.885555 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:29.885553 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs podName:6fe0d1a0-3adb-4b33-89b6-1f39fb282399 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:31.885538393 +0000 UTC m=+4.873878861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs") pod "network-metrics-daemon-z4mc4" (UID: "6fe0d1a0-3adb-4b33-89b6-1f39fb282399") : secret "metrics-daemon-secret" not found Oct 30 17:56:29.891958 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.891938 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b26b3373-a3b2-4a0d-bc89-94bd729b849e-metrics-tls\") pod \"dns-default-867jv\" (UID: \"b26b3373-a3b2-4a0d-bc89-94bd729b849e\") " pod="openshift-dns/dns-default-867jv" Oct 30 17:56:29.953894 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:29.953872 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-867jv" Oct 30 17:56:30.003230 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:30.003207 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:30.206110 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:30.206087 5014 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:56:30.246575 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:30.246546 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-867jv"] Oct 30 17:56:30.249364 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:30.249338 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26b3373_a3b2_4a0d_bc89_94bd729b849e.slice/crio-affbc00ab582239ed956958e7de156119b70ec1992139a7f1f21def6fa385e0b WatchSource:0}: Error finding container affbc00ab582239ed956958e7de156119b70ec1992139a7f1f21def6fa385e0b: Status 404 returned error can't find the container with id affbc00ab582239ed956958e7de156119b70ec1992139a7f1f21def6fa385e0b Oct 30 17:56:30.251388 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:30.251371 5014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:56:30.296486 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:30.296469 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wv5gn" Oct 30 17:56:31.212274 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.212247 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-867jv" event={"ID":"b26b3373-a3b2-4a0d-bc89-94bd729b849e","Type":"ContainerStarted","Data":"affbc00ab582239ed956958e7de156119b70ec1992139a7f1f21def6fa385e0b"} Oct 30 17:56:31.213789 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.213767 5014 generic.go:356] "Generic (PLEG): container finished" podID="e3a51993-67ad-4e39-85fa-0f75e7cbe5b8" containerID="dd0ccb667b2a48be5c86edebc487e3f29ea4ba429029ac70a4fec2baf8eac170" exitCode=0 Oct 30 17:56:31.213866 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.213840 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" event={"ID":"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8","Type":"ContainerDied","Data":"dd0ccb667b2a48be5c86edebc487e3f29ea4ba429029ac70a4fec2baf8eac170"} Oct 30 17:56:31.214268 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.214253 5014 scope.go:122] "RemoveContainer" containerID="dd0ccb667b2a48be5c86edebc487e3f29ea4ba429029ac70a4fec2baf8eac170" Oct 30 17:56:31.564953 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.564929 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:31.565187 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.565161 5014 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:56:31.906813 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.906785 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:31.906913 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.906824 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:31.906913 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.906854 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:31.906913 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.906883 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:31.906913 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.906904 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:31.907104 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.906935 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:31.907104 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:31.906962 5014 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:56:31.907104 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:31.907035 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls podName:85b62a96-cc23-4b22-ade6-5066759bbec6 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:35.907013794 +0000 UTC m=+8.895354278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9gjwg" (UID: "85b62a96-cc23-4b22-ade6-5066759bbec6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:56:31.907104 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:31.907044 5014 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:31.907104 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:31.907054 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:35.907044829 +0000 UTC m=+8.895385294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : configmap references non-existent config key: service-ca.crt Oct 30 17:56:31.907104 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:31.907074 5014 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:56:31.907104 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:31.907100 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls podName:de9d2339-d6be-4332-921d-ec5717b06699 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:35.907083636 +0000 UTC m=+8.895424121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wb2x4" (UID: "de9d2339-d6be-4332-921d-ec5717b06699") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:56:31.907417 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:31.907127 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs podName:6fe0d1a0-3adb-4b33-89b6-1f39fb282399 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:35.907111134 +0000 UTC m=+8.895451611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs") pod "network-metrics-daemon-z4mc4" (UID: "6fe0d1a0-3adb-4b33-89b6-1f39fb282399") : secret "metrics-daemon-secret" not found Oct 30 17:56:31.909301 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.909272 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f53a361f-64db-41a3-abd1-c04251831048-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htdwd\" (UID: \"f53a361f-64db-41a3-abd1-c04251831048\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:31.909398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:31.909382 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27d2c28b-a2b2-4293-ad25-a0e69b939b9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgr98\" (UID: \"27d2c28b-a2b2-4293-ad25-a0e69b939b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:32.045987 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.045962 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" Oct 30 17:56:32.048106 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.048083 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" Oct 30 17:56:32.219158 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.219130 5014 generic.go:356] "Generic (PLEG): container finished" podID="f2070749-ec46-433b-975e-470c734e5fe5" containerID="b208431de6eea7c9d9fd3b48eb2cea41a147fc1351ef508ce0fceed55042fd6b" exitCode=0 Oct 30 17:56:32.219541 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.219206 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j4bgx" event={"ID":"f2070749-ec46-433b-975e-470c734e5fe5","Type":"ContainerDied","Data":"b208431de6eea7c9d9fd3b48eb2cea41a147fc1351ef508ce0fceed55042fd6b"} Oct 30 17:56:32.219631 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.219588 5014 scope.go:122] "RemoveContainer" containerID="b208431de6eea7c9d9fd3b48eb2cea41a147fc1351ef508ce0fceed55042fd6b" Oct 30 17:56:32.221269 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.221245 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-867jv" event={"ID":"b26b3373-a3b2-4a0d-bc89-94bd729b849e","Type":"ContainerStarted","Data":"ae8569c2eb764ecdb737562a02ed152bb9bb6c9210022e4602e0586de6d85586"} Oct 30 17:56:32.221432 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.221297 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-867jv" event={"ID":"b26b3373-a3b2-4a0d-bc89-94bd729b849e","Type":"ContainerStarted","Data":"ed1cfaaad0109adac0beb7a121a01229eb7a86c630fba497736eb9e8cb8f77c3"} Oct 30 17:56:32.223297 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.223277 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jg6bj" event={"ID":"e3a51993-67ad-4e39-85fa-0f75e7cbe5b8","Type":"ContainerStarted","Data":"410d088bbee53d44db2c6033db32139aa85738f144e02b0f6442fc45b188edcb"} Oct 30 17:56:32.347981 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.347951 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98"] Oct 30 17:56:32.356201 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.356174 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-htdwd"] Oct 30 17:56:32.359872 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:32.359813 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53a361f_64db_41a3_abd1_c04251831048.slice/crio-a6f8659578d027739b478dce73290240db5ac8853bbae77aa42ef80671a39c54 WatchSource:0}: Error finding container a6f8659578d027739b478dce73290240db5ac8853bbae77aa42ef80671a39c54: Status 404 returned error can't find the container with id a6f8659578d027739b478dce73290240db5ac8853bbae77aa42ef80671a39c54 Oct 30 17:56:32.625324 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.625258 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:32.625458 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.625402 5014 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:56:32.630082 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.630058 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5mn6l" Oct 30 17:56:32.698548 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:32.698528 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-867jv" Oct 30 17:56:33.229095 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:33.229056 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j4bgx" event={"ID":"f2070749-ec46-433b-975e-470c734e5fe5","Type":"ContainerStarted","Data":"a67efcf5dd7ccfb326536bcd2a4b5d90b397d5adf41eed9bf66c019a9cd7533f"} Oct 30 17:56:33.230453 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:33.230426 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" event={"ID":"f53a361f-64db-41a3-abd1-c04251831048","Type":"ContainerStarted","Data":"a6f8659578d027739b478dce73290240db5ac8853bbae77aa42ef80671a39c54"} Oct 30 17:56:33.231973 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:33.231945 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" event={"ID":"27d2c28b-a2b2-4293-ad25-a0e69b939b9c","Type":"ContainerStarted","Data":"96a703f96ffeafbac4bbcf3dd994ca4d37b81517a1dd474162d10d7c174fa96d"} Oct 30 17:56:34.237307 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:34.237262 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-htdwd" event={"ID":"f53a361f-64db-41a3-abd1-c04251831048","Type":"ContainerStarted","Data":"ba24a571852147c39d52eb36a484d3e5bf82b56094eab5dab4eac142d2879441"} Oct 30 17:56:34.239105 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:34.239076 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" event={"ID":"27d2c28b-a2b2-4293-ad25-a0e69b939b9c","Type":"ContainerStarted","Data":"fd117236665bf2d8aafb34d0c418f6857feb8a83abe3bfd623d322972e08f680"} Oct 30 17:56:35.133687 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:35.133658 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:35.243915 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:35.243885 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgr98" event={"ID":"27d2c28b-a2b2-4293-ad25-a0e69b939b9c","Type":"ContainerStarted","Data":"1e535669c0b4dd698ba1ef1f35f9d8938b6b4b64a4e42105b730cb300d06a6a7"} Oct 30 17:56:35.945913 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:35.945877 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:35.945913 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:35.945919 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:35.946165 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:35.945949 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:35.946165 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:35.945992 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:35.946165 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:35.946159 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle podName:2b407f39-23cc-432f-a5d7-3a377e4762a5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:43.946137971 +0000 UTC m=+16.934478453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle") pod "router-default-c5cf884d7-2k5bz" (UID: "2b407f39-23cc-432f-a5d7-3a377e4762a5") : configmap references non-existent config key: service-ca.crt Oct 30 17:56:35.948799 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:35.948769 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe0d1a0-3adb-4b33-89b6-1f39fb282399-metrics-certs\") pod \"network-metrics-daemon-z4mc4\" (UID: \"6fe0d1a0-3adb-4b33-89b6-1f39fb282399\") " pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:35.948927 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:35.948893 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b62a96-cc23-4b22-ade6-5066759bbec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9gjwg\" (UID: \"85b62a96-cc23-4b22-ade6-5066759bbec6\") " pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:35.949034 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:35.949006 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de9d2339-d6be-4332-921d-ec5717b06699-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wb2x4\" (UID: \"de9d2339-d6be-4332-921d-ec5717b06699\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:35.951751 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:35.951732 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4mc4" Oct 30 17:56:35.957933 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:35.957907 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9gjwg" Oct 30 17:56:36.246609 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:36.246530 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" Oct 30 17:56:36.254044 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:36.254020 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z4mc4"] Oct 30 17:56:36.256970 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:36.256945 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe0d1a0_3adb_4b33_89b6_1f39fb282399.slice/crio-cb6ea41b8aa5862cf42ba61c600fff1ac773c9ceef70497bad84603b1ab07195 WatchSource:0}: Error finding container cb6ea41b8aa5862cf42ba61c600fff1ac773c9ceef70497bad84603b1ab07195: Status 404 returned error can't find the container with id cb6ea41b8aa5862cf42ba61c600fff1ac773c9ceef70497bad84603b1ab07195 Oct 30 17:56:36.266281 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:36.266262 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9gjwg"] Oct 30 17:56:36.268705 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:36.268670 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b62a96_cc23_4b22_ade6_5066759bbec6.slice/crio-8eb6e41351c31d68028b838b0b4f680ac9e456dd411e6fb46fa9b021f8c41b06 WatchSource:0}: Error finding container 8eb6e41351c31d68028b838b0b4f680ac9e456dd411e6fb46fa9b021f8c41b06: Status 404 returned error can't find the container with id 8eb6e41351c31d68028b838b0b4f680ac9e456dd411e6fb46fa9b021f8c41b06 Oct 30 17:56:36.543506 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:36.543448 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4"] Oct 30 17:56:36.547007 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:36.546983 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9d2339_d6be_4332_921d_ec5717b06699.slice/crio-2ff88da1bcec11df04ae5bca55034706f34dbfca87e01989c0dccf0312842b41 WatchSource:0}: Error finding container 2ff88da1bcec11df04ae5bca55034706f34dbfca87e01989c0dccf0312842b41: Status 404 returned error can't find the container with id 2ff88da1bcec11df04ae5bca55034706f34dbfca87e01989c0dccf0312842b41 Oct 30 17:56:36.959907 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:36.959885 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:37.258468 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:37.258430 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z4mc4" event={"ID":"6fe0d1a0-3adb-4b33-89b6-1f39fb282399","Type":"ContainerStarted","Data":"02b81f313b863c9c48db30838850e483e58e8801bdec1ce54455265d00a72454"} Oct 30 17:56:37.258895 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:37.258502 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z4mc4" event={"ID":"6fe0d1a0-3adb-4b33-89b6-1f39fb282399","Type":"ContainerStarted","Data":"cb6ea41b8aa5862cf42ba61c600fff1ac773c9ceef70497bad84603b1ab07195"} Oct 30 17:56:37.261139 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:37.261113 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" event={"ID":"de9d2339-d6be-4332-921d-ec5717b06699","Type":"ContainerStarted","Data":"2ff88da1bcec11df04ae5bca55034706f34dbfca87e01989c0dccf0312842b41"} Oct 30 17:56:37.263995 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:37.263968 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9gjwg" event={"ID":"85b62a96-cc23-4b22-ade6-5066759bbec6","Type":"ContainerStarted","Data":"443aabb07cd3d7eb83e1059c94a67878aa0195e4180a40c04a58d04c8b740cb7"} Oct 30 17:56:37.264083 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:37.264024 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9gjwg" event={"ID":"85b62a96-cc23-4b22-ade6-5066759bbec6","Type":"ContainerStarted","Data":"0e5aeb8dd68e88b198807188c003b355a92e1950f71784d3159ed74d2f901f33"} Oct 30 17:56:37.264083 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:37.264055 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9gjwg" event={"ID":"85b62a96-cc23-4b22-ade6-5066759bbec6","Type":"ContainerStarted","Data":"8eb6e41351c31d68028b838b0b4f680ac9e456dd411e6fb46fa9b021f8c41b06"} Oct 30 17:56:38.271765 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:38.271724 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z4mc4" event={"ID":"6fe0d1a0-3adb-4b33-89b6-1f39fb282399","Type":"ContainerStarted","Data":"502ae31832e9238909d43b7f45d70c434c63cd52aaa0681caa5c34904b181bd1"} Oct 30 17:56:38.273650 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:38.273623 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wb2x4" event={"ID":"de9d2339-d6be-4332-921d-ec5717b06699","Type":"ContainerStarted","Data":"053573e43d6d9317f1ba5856707f0b492c193681a81ed2c78c6d20f8988d7631"} Oct 30 17:56:39.280524 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:39.280484 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9gjwg" event={"ID":"85b62a96-cc23-4b22-ade6-5066759bbec6","Type":"ContainerStarted","Data":"c298289ce651f1670bcb9dd3cf16be48281ffa75a8f1bbd1580ac3ed16a42739"} Oct 30 17:56:43.241476 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:43.241446 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-867jv" Oct 30 17:56:43.332143 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:56:43.332109 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 17:56:44.007187 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:44.007156 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:44.007786 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:44.007764 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b407f39-23cc-432f-a5d7-3a377e4762a5-service-ca-bundle\") pod \"router-default-c5cf884d7-2k5bz\" (UID: \"2b407f39-23cc-432f-a5d7-3a377e4762a5\") " pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:44.042919 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:44.042889 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:44.351867 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:44.351835 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c5cf884d7-2k5bz"] Oct 30 17:56:44.354994 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:44.354972 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b407f39_23cc_432f_a5d7_3a377e4762a5.slice/crio-2278ec1dd98c15d6f74a0080f3a0bdd0c2407555881e31ba150dfedc6d02ebc2 WatchSource:0}: Error finding container 2278ec1dd98c15d6f74a0080f3a0bdd0c2407555881e31ba150dfedc6d02ebc2: Status 404 returned error can't find the container with id 2278ec1dd98c15d6f74a0080f3a0bdd0c2407555881e31ba150dfedc6d02ebc2 Oct 30 17:56:45.025467 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:45.025424 5014 patch_prober.go:29] interesting pod/image-registry-698f87866c-k4grf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 17:56:45.025713 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:45.025488 5014 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-698f87866c-k4grf" podUID="78c06084-1bed-46c5-a576-f6129e21e062" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:56:45.137993 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:45.137952 5014 patch_prober.go:29] interesting pod/image-registry-5b849f747d-p4c4m container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 17:56:45.138110 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:45.138042 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" podUID="1136549d-7491-452c-b396-f839668b5a19" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:56:45.305714 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:45.305680 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c5cf884d7-2k5bz" event={"ID":"2b407f39-23cc-432f-a5d7-3a377e4762a5","Type":"ContainerStarted","Data":"623518b555c23b3288c8ad7b9cea27411376ddf71297b01b85908ed500bf0309"} Oct 30 17:56:45.305835 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:45.305723 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c5cf884d7-2k5bz" event={"ID":"2b407f39-23cc-432f-a5d7-3a377e4762a5","Type":"ContainerStarted","Data":"2278ec1dd98c15d6f74a0080f3a0bdd0c2407555881e31ba150dfedc6d02ebc2"} Oct 30 17:56:46.044100 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.044069 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:46.046916 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.046890 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:46.310123 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.310087 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:46.311286 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.311261 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c5cf884d7-2k5bz" Oct 30 17:56:46.479133 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.479100 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bnqm4-s2tbr"] Oct 30 17:56:46.483346 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.483320 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.505732 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.505707 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-6r46n"] Oct 30 17:56:46.509406 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.509385 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn"] Oct 30 17:56:46.509541 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.509526 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6r46n" Oct 30 17:56:46.513255 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.513235 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bnqm4-s2tbr"] Oct 30 17:56:46.513356 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.513338 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn" Oct 30 17:56:46.539557 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.539536 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 17:56:46.539693 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.539582 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 17:56:46.539923 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.539895 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-s2svl\"" Oct 30 17:56:46.543356 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.543336 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 17:56:46.543517 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.543496 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b849f747d-p4c4m"] Oct 30 17:56:46.543710 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.543689 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qt9cq\"" Oct 30 17:56:46.549010 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.548987 5014 patch_prober.go:29] interesting pod/image-registry-5b849f747d-p4c4m container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 17:56:46.549103 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.549028 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" podUID="1136549d-7491-452c-b396-f839668b5a19" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:56:46.551384 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.551364 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6r46n"] Oct 30 17:56:46.552811 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.552793 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn"] Oct 30 17:56:46.605968 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.605911 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b6577d447-5pww9"] Oct 30 17:56:46.609200 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.609181 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.628523 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.628504 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2gv8\" (UniqueName: \"kubernetes.io/projected/d21419a6-e9b0-46d9-97b8-4484fc94c0dd-kube-api-access-r2gv8\") pod \"downloads-6dd789c8b7-6r46n\" (UID: \"d21419a6-e9b0-46d9-97b8-4484fc94c0dd\") " pod="openshift-console/downloads-6dd789c8b7-6r46n" Oct 30 17:56:46.628643 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.628537 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-archives-path\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.628643 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.628563 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb477009-3d76-4f94-a25c-3d1320684c68-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-m5hwn\" (UID: \"eb477009-3d76-4f94-a25c-3d1320684c68\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn" Oct 30 17:56:46.628727 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.628694 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4r79\" (UniqueName: \"kubernetes.io/projected/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-kube-api-access-g4r79\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.628764 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.628729 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-trusted-ca-bundle\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.628764 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.628750 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d21419a6-e9b0-46d9-97b8-4484fc94c0dd-tmp\") pod \"downloads-6dd789c8b7-6r46n\" (UID: \"d21419a6-e9b0-46d9-97b8-4484fc94c0dd\") " pod="openshift-console/downloads-6dd789c8b7-6r46n" Oct 30 17:56:46.628846 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.628833 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-service-ca-bundle\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.645002 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.644983 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b6577d447-5pww9"] Oct 30 17:56:46.729558 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729537 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2gv8\" (UniqueName: \"kubernetes.io/projected/d21419a6-e9b0-46d9-97b8-4484fc94c0dd-kube-api-access-r2gv8\") pod \"downloads-6dd789c8b7-6r46n\" (UID: \"d21419a6-e9b0-46d9-97b8-4484fc94c0dd\") " pod="openshift-console/downloads-6dd789c8b7-6r46n" Oct 30 17:56:46.729682 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729568 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lqkb\" (UniqueName: \"kubernetes.io/projected/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-kube-api-access-4lqkb\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.729682 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729607 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-archives-path\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.729682 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729634 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb477009-3d76-4f94-a25c-3d1320684c68-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-m5hwn\" (UID: \"eb477009-3d76-4f94-a25c-3d1320684c68\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn" Oct 30 17:56:46.729788 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729696 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4r79\" (UniqueName: \"kubernetes.io/projected/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-kube-api-access-g4r79\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.729788 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729737 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-trusted-ca-bundle\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.729788 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729757 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-image-registry-private-configuration\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.729788 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729778 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d21419a6-e9b0-46d9-97b8-4484fc94c0dd-tmp\") pod \"downloads-6dd789c8b7-6r46n\" (UID: \"d21419a6-e9b0-46d9-97b8-4484fc94c0dd\") " pod="openshift-console/downloads-6dd789c8b7-6r46n" Oct 30 17:56:46.729975 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729795 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-registry-tls\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.729975 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729904 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-installation-pull-secrets\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.729975 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729950 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-bound-sa-token\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.730136 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.729989 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-ca-trust-extracted\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.730136 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.730013 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-trusted-ca\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.730136 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.730105 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d21419a6-e9b0-46d9-97b8-4484fc94c0dd-tmp\") pod \"downloads-6dd789c8b7-6r46n\" (UID: \"d21419a6-e9b0-46d9-97b8-4484fc94c0dd\") " pod="openshift-console/downloads-6dd789c8b7-6r46n" Oct 30 17:56:46.730296 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.730141 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-service-ca-bundle\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.730296 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.730219 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-registry-certificates\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.730401 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.730373 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-archives-path\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.730830 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.730808 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-trusted-ca-bundle\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.730879 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.730836 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-service-ca-bundle\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.732444 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.732423 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb477009-3d76-4f94-a25c-3d1320684c68-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-m5hwn\" (UID: \"eb477009-3d76-4f94-a25c-3d1320684c68\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn" Oct 30 17:56:46.755008 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.754987 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4r79\" (UniqueName: \"kubernetes.io/projected/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-kube-api-access-g4r79\") pod \"periodic-gathering-bnqm4-s2tbr\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.755179 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.755162 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2gv8\" (UniqueName: \"kubernetes.io/projected/d21419a6-e9b0-46d9-97b8-4484fc94c0dd-kube-api-access-r2gv8\") pod \"downloads-6dd789c8b7-6r46n\" (UID: \"d21419a6-e9b0-46d9-97b8-4484fc94c0dd\") " pod="openshift-console/downloads-6dd789c8b7-6r46n" Oct 30 17:56:46.799646 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.799628 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:56:46.825939 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.825906 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6r46n" Oct 30 17:56:46.834901 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.834137 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn" Oct 30 17:56:46.834901 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.834528 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-installation-pull-secrets\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.834901 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.834582 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-bound-sa-token\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.834901 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.834663 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-ca-trust-extracted\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.834901 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.834698 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-trusted-ca\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.834901 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.834741 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-registry-certificates\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.834901 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.834784 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lqkb\" (UniqueName: \"kubernetes.io/projected/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-kube-api-access-4lqkb\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.834901 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.834859 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-image-registry-private-configuration\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.834901 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.834895 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-registry-tls\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.837254 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.837234 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-ca-trust-extracted\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.837907 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.837808 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-registry-certificates\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.838274 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.838251 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-installation-pull-secrets\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.838771 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.838748 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-trusted-ca\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.839727 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.839695 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-registry-tls\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.839798 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.839733 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-image-registry-private-configuration\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.863389 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.862663 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lqkb\" (UniqueName: \"kubernetes.io/projected/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-kube-api-access-4lqkb\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.863389 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.863202 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afb40c3a-eec0-41f8-b406-b5bc75ac66ac-bound-sa-token\") pod \"image-registry-6b6577d447-5pww9\" (UID: \"afb40c3a-eec0-41f8-b406-b5bc75ac66ac\") " pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:46.920808 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:46.920624 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:47.095811 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.095789 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bnqm4-s2tbr"] Oct 30 17:56:47.098651 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:47.098625 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6cb5c2_6a10_41ef_8d2e_aa337d19995f.slice/crio-804e11028a5d3846963acaea2bf6064759fb6a033303bfa29ee8e98b437e736d WatchSource:0}: Error finding container 804e11028a5d3846963acaea2bf6064759fb6a033303bfa29ee8e98b437e736d: Status 404 returned error can't find the container with id 804e11028a5d3846963acaea2bf6064759fb6a033303bfa29ee8e98b437e736d Oct 30 17:56:47.146143 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.146103 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6r46n"] Oct 30 17:56:47.150112 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:47.150086 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21419a6_e9b0_46d9_97b8_4484fc94c0dd.slice/crio-6253908b19cec64c0be490f006829c006e79f3e87adf7114a125d6a5ef85c417 WatchSource:0}: Error finding container 6253908b19cec64c0be490f006829c006e79f3e87adf7114a125d6a5ef85c417: Status 404 returned error can't find the container with id 6253908b19cec64c0be490f006829c006e79f3e87adf7114a125d6a5ef85c417 Oct 30 17:56:47.169202 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.169180 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn"] Oct 30 17:56:47.172128 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:47.172102 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb477009_3d76_4f94_a25c_3d1320684c68.slice/crio-8735113bd304022d0ca60ba18ef15f583894fae731c5a55dc56388cf9e60a6e9 WatchSource:0}: Error finding container 8735113bd304022d0ca60ba18ef15f583894fae731c5a55dc56388cf9e60a6e9: Status 404 returned error can't find the container with id 8735113bd304022d0ca60ba18ef15f583894fae731c5a55dc56388cf9e60a6e9 Oct 30 17:56:47.231481 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.231454 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b6577d447-5pww9"] Oct 30 17:56:47.234655 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:47.234633 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb40c3a_eec0_41f8_b406_b5bc75ac66ac.slice/crio-2214a431501b0ea005d4a4110c918f13c3d08c08c294fc2c753ec9cc6b5ee309 WatchSource:0}: Error finding container 2214a431501b0ea005d4a4110c918f13c3d08c08c294fc2c753ec9cc6b5ee309: Status 404 returned error can't find the container with id 2214a431501b0ea005d4a4110c918f13c3d08c08c294fc2c753ec9cc6b5ee309 Oct 30 17:56:47.317288 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.317251 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" event={"ID":"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f","Type":"ContainerStarted","Data":"c5a1c41d53d53371d6865b78ad1f11cec60dac6baf3736aa32e9199086b56b53"} Oct 30 17:56:47.317419 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.317307 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" event={"ID":"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f","Type":"ContainerStarted","Data":"804e11028a5d3846963acaea2bf6064759fb6a033303bfa29ee8e98b437e736d"} Oct 30 17:56:47.318660 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.318634 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6r46n" event={"ID":"d21419a6-e9b0-46d9-97b8-4484fc94c0dd","Type":"ContainerStarted","Data":"6253908b19cec64c0be490f006829c006e79f3e87adf7114a125d6a5ef85c417"} Oct 30 17:56:47.320183 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.320163 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6577d447-5pww9" event={"ID":"afb40c3a-eec0-41f8-b406-b5bc75ac66ac","Type":"ContainerStarted","Data":"ad1c52a58fd23abc6067b8047a3cfd28fabac8d47bb41f25e315cca72c6579d0"} Oct 30 17:56:47.320269 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.320197 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6577d447-5pww9" event={"ID":"afb40c3a-eec0-41f8-b406-b5bc75ac66ac","Type":"ContainerStarted","Data":"2214a431501b0ea005d4a4110c918f13c3d08c08c294fc2c753ec9cc6b5ee309"} Oct 30 17:56:47.320577 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.320563 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:56:47.321947 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.321926 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn" event={"ID":"eb477009-3d76-4f94-a25c-3d1320684c68","Type":"ContainerStarted","Data":"8735113bd304022d0ca60ba18ef15f583894fae731c5a55dc56388cf9e60a6e9"} Oct 30 17:56:47.356529 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.356474 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" podStartSLOduration=1.356463242 podStartE2EDuration="1.356463242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:47.355577469 +0000 UTC m=+20.343917956" watchObservedRunningTime="2026-10-30 17:56:47.356463242 +0000 UTC m=+20.344803729" Oct 30 17:56:47.406944 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:47.406864 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b6577d447-5pww9" podStartSLOduration=1.406851221 podStartE2EDuration="1.406851221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:47.405695653 +0000 UTC m=+20.394036158" watchObservedRunningTime="2026-10-30 17:56:47.406851221 +0000 UTC m=+20.395191707" Oct 30 17:56:48.336218 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:48.336163 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn" event={"ID":"eb477009-3d76-4f94-a25c-3d1320684c68","Type":"ContainerStarted","Data":"3cf74316c3ff80cd7390fe140850ad8ceab9c56b072c4b2938b483fb4fb272bb"} Oct 30 17:56:48.337620 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:48.337246 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn" Oct 30 17:56:48.342586 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:48.338657 5014 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-m5hwn container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": dial tcp 10.132.0.30:8443: connect: connection refused" start-of-body= Oct 30 17:56:48.342586 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:48.338727 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn" podUID="eb477009-3d76-4f94-a25c-3d1320684c68" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": dial tcp 10.132.0.30:8443: connect: connection refused" Oct 30 17:56:48.376095 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:48.375875 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn" podStartSLOduration=1.308903899 podStartE2EDuration="2.37585871s" totalImagesPullingTime="1.066954811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:48.37548718 +0000 UTC m=+21.363827666" watchObservedRunningTime="2026-10-30 17:56:48.37585871 +0000 UTC m=+21.364199634" Oct 30 17:56:49.344776 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:49.344745 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-m5hwn" Oct 30 17:56:49.383640 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:49.383585 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-htdwd_f53a361f-64db-41a3-abd1-c04251831048/networking-console-plugin/0.log" Oct 30 17:56:49.583925 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:49.583894 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 17:56:49.785628 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:49.785538 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/2.log" Oct 30 17:56:50.591168 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:50.591144 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-867jv_b26b3373-a3b2-4a0d-bc89-94bd729b849e/dns/0.log" Oct 30 17:56:50.783328 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:50.783290 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-867jv_b26b3373-a3b2-4a0d-bc89-94bd729b849e/kube-rbac-proxy/0.log" Oct 30 17:56:50.983559 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:50.983487 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-94ttz_f1043899-80a3-47a9-9d1c-0421cd68df00/dns-node-resolver/0.log" Oct 30 17:56:51.219986 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:51.219961 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:56:51.583122 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:51.583093 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5b849f747d-p4c4m_1136549d-7491-452c-b396-f839668b5a19/registry/0.log" Oct 30 17:56:51.783393 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:51.783367 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-698f87866c-k4grf_78c06084-1bed-46c5-a576-f6129e21e062/registry/0.log" Oct 30 17:56:51.983645 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:51.983551 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b6577d447-5pww9_afb40c3a-eec0-41f8-b406-b5bc75ac66ac/registry/0.log" Oct 30 17:56:52.183439 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:52.183414 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l67qw_f4572363-4113-4e99-a34a-1b33a4cc08ff/node-ca/0.log" Oct 30 17:56:52.784712 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:52.784680 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c5cf884d7-2k5bz_2b407f39-23cc-432f-a5d7-3a377e4762a5/router/0.log" Oct 30 17:56:53.183069 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:53.183030 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vj5k4_de504e55-5beb-4f9a-a08a-b4f4e1f999d6/serve-healthcheck-canary/0.log" Oct 30 17:56:54.582424 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:54.582392 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddxgl" Oct 30 17:56:55.152668 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.152634 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gvhjk"] Oct 30 17:56:55.189606 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.189565 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.203440 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.203420 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 17:56:55.203923 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.203891 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 17:56:55.204118 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.204099 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:56:55.204226 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.204212 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 17:56:55.204411 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.204389 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cclgx\"" Oct 30 17:56:55.314665 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.314633 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.314825 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.314669 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d504699e-3ab5-4169-9902-5be32fc3abdd-root\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.314825 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.314708 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d504699e-3ab5-4169-9902-5be32fc3abdd-metrics-client-ca\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.314932 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.314826 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d504699e-3ab5-4169-9902-5be32fc3abdd-sys\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.314932 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.314865 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8p6\" (UniqueName: \"kubernetes.io/projected/d504699e-3ab5-4169-9902-5be32fc3abdd-kube-api-access-9t8p6\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.314997 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.314938 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-textfile\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.314997 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.314990 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-tls\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.315077 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.315005 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-wtmp\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.315115 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.315076 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416369 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416293 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-tls\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416369 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416334 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-wtmp\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416586 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416366 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416586 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416418 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416586 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416447 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d504699e-3ab5-4169-9902-5be32fc3abdd-root\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416586 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416488 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d504699e-3ab5-4169-9902-5be32fc3abdd-metrics-client-ca\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416586 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416530 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d504699e-3ab5-4169-9902-5be32fc3abdd-sys\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416586 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416558 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8p6\" (UniqueName: \"kubernetes.io/projected/d504699e-3ab5-4169-9902-5be32fc3abdd-kube-api-access-9t8p6\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416586 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416581 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-textfile\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416974 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416627 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-wtmp\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416974 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416736 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d504699e-3ab5-4169-9902-5be32fc3abdd-root\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.416974 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.416773 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d504699e-3ab5-4169-9902-5be32fc3abdd-sys\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.417345 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.417319 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-textfile\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.417512 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.417492 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.417608 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.417530 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d504699e-3ab5-4169-9902-5be32fc3abdd-metrics-client-ca\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.419128 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.419110 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-tls\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.423183 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.423161 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d504699e-3ab5-4169-9902-5be32fc3abdd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.442809 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.442785 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8p6\" (UniqueName: \"kubernetes.io/projected/d504699e-3ab5-4169-9902-5be32fc3abdd-kube-api-access-9t8p6\") pod \"node-exporter-gvhjk\" (UID: \"d504699e-3ab5-4169-9902-5be32fc3abdd\") " pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.552106 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:55.552075 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gvhjk" Oct 30 17:56:55.583964 ip-10-0-133-62 kubenswrapper[5014]: W1030 17:56:55.583933 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd504699e_3ab5_4169_9902_5be32fc3abdd.slice/crio-c14ff7ca8333b813d77720c1b0b189e49aaacff452f6451705eabba03647bae2 WatchSource:0}: Error finding container c14ff7ca8333b813d77720c1b0b189e49aaacff452f6451705eabba03647bae2: Status 404 returned error can't find the container with id c14ff7ca8333b813d77720c1b0b189e49aaacff452f6451705eabba03647bae2 Oct 30 17:56:56.380549 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:56.380467 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvhjk" event={"ID":"d504699e-3ab5-4169-9902-5be32fc3abdd","Type":"ContainerStarted","Data":"c14ff7ca8333b813d77720c1b0b189e49aaacff452f6451705eabba03647bae2"} Oct 30 17:56:56.549240 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:56.549213 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:56:56.963411 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:56.963377 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lzl5g" Oct 30 17:56:57.387519 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:57.387478 5014 generic.go:356] "Generic (PLEG): container finished" podID="d504699e-3ab5-4169-9902-5be32fc3abdd" containerID="a1e8cfc04f7f3cd70563c4a49c46e3bcc7f87c4465581e9a490c1efa2b68cd9c" exitCode=0 Oct 30 17:56:57.387717 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:57.387617 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvhjk" event={"ID":"d504699e-3ab5-4169-9902-5be32fc3abdd","Type":"ContainerDied","Data":"a1e8cfc04f7f3cd70563c4a49c46e3bcc7f87c4465581e9a490c1efa2b68cd9c"} Oct 30 17:56:58.419779 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:58.419740 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvhjk" event={"ID":"d504699e-3ab5-4169-9902-5be32fc3abdd","Type":"ContainerStarted","Data":"46128170dee1c7c8019a256f8ede9cd9223adfd7f3c21d7e4026d75c63113edf"} Oct 30 17:56:58.420246 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:58.419807 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvhjk" event={"ID":"d504699e-3ab5-4169-9902-5be32fc3abdd","Type":"ContainerStarted","Data":"926aad51c29a72b0057d5387331de4b3c1c2789b23c7a81b900c26f2883516d4"} Oct 30 17:56:58.464225 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:56:58.463992 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gvhjk" podStartSLOduration=1.95250683 podStartE2EDuration="3.463975574s" totalImagesPullingTime="1.418714062s" totalInitContainerRuntime="92.754682ms" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:58.462288748 +0000 UTC m=+31.450629236" watchObservedRunningTime="2026-10-30 17:56:58.463975574 +0000 UTC m=+31.452316059" Oct 30 17:57:03.444353 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:03.444312 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6r46n" event={"ID":"d21419a6-e9b0-46d9-97b8-4484fc94c0dd","Type":"ContainerStarted","Data":"c8fc256d3ea36cff442d2396b8fee000d34292d5e3c005a99df80e5e16bb011c"} Oct 30 17:57:03.445398 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:03.445378 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-6r46n" Oct 30 17:57:03.446941 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:03.446920 5014 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-6r46n container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body= Oct 30 17:57:03.446988 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:03.446966 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-6r46n" podUID="d21419a6-e9b0-46d9-97b8-4484fc94c0dd" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" Oct 30 17:57:03.504314 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:03.504252 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-6r46n" podStartSLOduration=1.319360302 podStartE2EDuration="17.504236687s" totalImagesPullingTime="16.184876385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:03.503378631 +0000 UTC m=+36.491719152" watchObservedRunningTime="2026-10-30 17:57:03.504236687 +0000 UTC m=+36.492577174" Oct 30 17:57:04.464062 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:04.464032 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-6r46n" Oct 30 17:57:09.347821 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:09.347733 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b6577d447-5pww9" Oct 30 17:57:09.439628 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:09.438292 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-698f87866c-k4grf"] Oct 30 17:57:11.566069 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.566014 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" podUID="1136549d-7491-452c-b396-f839668b5a19" containerName="registry" containerID="cri-o://3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e" gracePeriod=30 Oct 30 17:57:11.872457 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.872421 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:57:11.977922 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.977894 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-trusted-ca\") pod \"1136549d-7491-452c-b396-f839668b5a19\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " Oct 30 17:57:11.978076 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.977927 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-registry-certificates\") pod \"1136549d-7491-452c-b396-f839668b5a19\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " Oct 30 17:57:11.978076 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.977988 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1136549d-7491-452c-b396-f839668b5a19-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1136549d-7491-452c-b396-f839668b5a19-ca-trust-extracted\") pod \"1136549d-7491-452c-b396-f839668b5a19\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " Oct 30 17:57:11.978076 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.978053 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-image-registry-private-configuration\") pod \"1136549d-7491-452c-b396-f839668b5a19\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " Oct 30 17:57:11.978237 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.978123 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-installation-pull-secrets\") pod \"1136549d-7491-452c-b396-f839668b5a19\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " Oct 30 17:57:11.978237 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.978156 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-kube-api-access-t5c28\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-kube-api-access-t5c28\") pod \"1136549d-7491-452c-b396-f839668b5a19\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " Oct 30 17:57:11.978237 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.978204 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\") pod \"1136549d-7491-452c-b396-f839668b5a19\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " Oct 30 17:57:11.978237 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.978231 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-bound-sa-token\") pod \"1136549d-7491-452c-b396-f839668b5a19\" (UID: \"1136549d-7491-452c-b396-f839668b5a19\") " Oct 30 17:57:11.978439 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.978324 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-trusted-ca" pod "1136549d-7491-452c-b396-f839668b5a19" (UID: "1136549d-7491-452c-b396-f839668b5a19"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:11.978439 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.978367 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-registry-certificates" pod "1136549d-7491-452c-b396-f839668b5a19" (UID: "1136549d-7491-452c-b396-f839668b5a19"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:11.978540 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.978512 5014 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-trusted-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:11.978540 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.978533 5014 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1136549d-7491-452c-b396-f839668b5a19-registry-certificates\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:11.981482 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.981455 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-bound-sa-token" pod "1136549d-7491-452c-b396-f839668b5a19" (UID: "1136549d-7491-452c-b396-f839668b5a19"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:11.981944 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.981850 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-installation-pull-secrets" pod "1136549d-7491-452c-b396-f839668b5a19" (UID: "1136549d-7491-452c-b396-f839668b5a19"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:11.982104 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.982077 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-image-registry-private-configuration" pod "1136549d-7491-452c-b396-f839668b5a19" (UID: "1136549d-7491-452c-b396-f839668b5a19"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:11.982212 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.982180 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-kube-api-access-t5c28" pod "1136549d-7491-452c-b396-f839668b5a19" (UID: "1136549d-7491-452c-b396-f839668b5a19"). InnerVolumeSpecName "kube-api-access-t5c28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:11.982212 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.982180 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls" pod "1136549d-7491-452c-b396-f839668b5a19" (UID: "1136549d-7491-452c-b396-f839668b5a19"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:11.986633 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:11.986577 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1136549d-7491-452c-b396-f839668b5a19-ca-trust-extracted" pod "1136549d-7491-452c-b396-f839668b5a19" (UID: "1136549d-7491-452c-b396-f839668b5a19"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:12.079547 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.079519 5014 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1136549d-7491-452c-b396-f839668b5a19-ca-trust-extracted\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:12.079547 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.079542 5014 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-image-registry-private-configuration\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:12.079547 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.079552 5014 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1136549d-7491-452c-b396-f839668b5a19-installation-pull-secrets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:12.079809 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.079561 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5c28\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-kube-api-access-t5c28\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:12.079809 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.079571 5014 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-registry-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:12.079809 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.079579 5014 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1136549d-7491-452c-b396-f839668b5a19-bound-sa-token\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:12.517194 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.517160 5014 generic.go:356] "Generic (PLEG): container finished" podID="1136549d-7491-452c-b396-f839668b5a19" containerID="3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e" exitCode=0 Oct 30 17:57:12.517356 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.517195 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" event={"ID":"1136549d-7491-452c-b396-f839668b5a19","Type":"ContainerDied","Data":"3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e"} Oct 30 17:57:12.517356 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.517248 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" event={"ID":"1136549d-7491-452c-b396-f839668b5a19","Type":"ContainerDied","Data":"6a895777390d7c2066ff707bac21dff7aa16c1ecfb285fdf0c5daca6da9907e2"} Oct 30 17:57:12.517356 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.517278 5014 scope.go:122] "RemoveContainer" containerID="3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e" Oct 30 17:57:12.517356 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.517208 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b849f747d-p4c4m" Oct 30 17:57:12.528675 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.528660 5014 scope.go:122] "RemoveContainer" containerID="3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e" Oct 30 17:57:12.528929 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:57:12.528907 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e\": container with ID starting with 3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e not found: ID does not exist" containerID="3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e" Oct 30 17:57:12.529003 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.528943 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e"} err="failed to get container status \"3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e\": rpc error: code = NotFound desc = could not find container \"3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e\": container with ID starting with 3ab3ea72b1b82962ec6e418f539ae012a23694040d798db3cb326b6ab8c63e6e not found: ID does not exist" Oct 30 17:57:12.577977 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.577955 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b849f747d-p4c4m"] Oct 30 17:57:12.589619 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:12.589581 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b849f747d-p4c4m"] Oct 30 17:57:13.141808 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:13.141779 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1136549d-7491-452c-b396-f839668b5a19" path="/var/lib/kubelet/pods/1136549d-7491-452c-b396-f839668b5a19/volumes" Oct 30 17:57:13.332424 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:57:13.332390 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 17:57:18.037099 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.037067 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fqlng-zdr42"] Oct 30 17:57:18.037617 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.037252 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-fqlng-zdr42" podUID="1a65a55c-27e9-4e39-a139-53ae907e0bfd" containerName="insights-gathering" containerID="cri-o://e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822" gracePeriod=30 Oct 30 17:57:18.342378 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.342349 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:57:18.426810 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.426785 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-service-ca-bundle\") pod \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " Oct 30 17:57:18.426931 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.426828 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-trusted-ca-bundle\") pod \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " Oct 30 17:57:18.426931 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.426901 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a65a55c-27e9-4e39-a139-53ae907e0bfd-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a65a55c-27e9-4e39-a139-53ae907e0bfd-archives-path\") pod \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " Oct 30 17:57:18.427002 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.426945 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a65a55c-27e9-4e39-a139-53ae907e0bfd-kube-api-access-8ll6l\" (UniqueName: \"kubernetes.io/projected/1a65a55c-27e9-4e39-a139-53ae907e0bfd-kube-api-access-8ll6l\") pod \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\" (UID: \"1a65a55c-27e9-4e39-a139-53ae907e0bfd\") " Oct 30 17:57:18.427118 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.427087 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-service-ca-bundle" pod "1a65a55c-27e9-4e39-a139-53ae907e0bfd" (UID: "1a65a55c-27e9-4e39-a139-53ae907e0bfd"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:18.427249 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.427225 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-trusted-ca-bundle" pod "1a65a55c-27e9-4e39-a139-53ae907e0bfd" (UID: "1a65a55c-27e9-4e39-a139-53ae907e0bfd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:18.427249 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.427233 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a65a55c-27e9-4e39-a139-53ae907e0bfd-archives-path" pod "1a65a55c-27e9-4e39-a139-53ae907e0bfd" (UID: "1a65a55c-27e9-4e39-a139-53ae907e0bfd"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:18.427249 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.427248 5014 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-service-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:18.429379 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.429359 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a65a55c-27e9-4e39-a139-53ae907e0bfd-kube-api-access-8ll6l" pod "1a65a55c-27e9-4e39-a139-53ae907e0bfd" (UID: "1a65a55c-27e9-4e39-a139-53ae907e0bfd"). InnerVolumeSpecName "kube-api-access-8ll6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:18.527802 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.527776 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ll6l\" (UniqueName: \"kubernetes.io/projected/1a65a55c-27e9-4e39-a139-53ae907e0bfd-kube-api-access-8ll6l\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:18.527802 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.527798 5014 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a65a55c-27e9-4e39-a139-53ae907e0bfd-trusted-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:18.527946 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.527809 5014 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a65a55c-27e9-4e39-a139-53ae907e0bfd-archives-path\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:18.545004 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.544970 5014 generic.go:356] "Generic (PLEG): container finished" podID="1a65a55c-27e9-4e39-a139-53ae907e0bfd" containerID="e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822" exitCode=2 Oct 30 17:57:18.545109 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.545016 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fqlng-zdr42" Oct 30 17:57:18.545192 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.545017 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fqlng-zdr42" event={"ID":"1a65a55c-27e9-4e39-a139-53ae907e0bfd","Type":"ContainerDied","Data":"e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822"} Oct 30 17:57:18.545192 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.545145 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fqlng-zdr42" event={"ID":"1a65a55c-27e9-4e39-a139-53ae907e0bfd","Type":"ContainerDied","Data":"cc7303134e85a69c59e996f3b5948a6cb5fc57ae411f334fbf7dc659bd3ecb2d"} Oct 30 17:57:18.545192 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.545167 5014 scope.go:122] "RemoveContainer" containerID="e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822" Oct 30 17:57:18.561715 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.561699 5014 scope.go:122] "RemoveContainer" containerID="e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822" Oct 30 17:57:18.561979 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:57:18.561961 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822\": container with ID starting with e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822 not found: ID does not exist" containerID="e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822" Oct 30 17:57:18.562029 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.561988 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822"} err="failed to get container status \"e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822\": rpc error: code = NotFound desc = could not find container \"e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822\": container with ID starting with e130c9415760189decdd610abc845e83b2de9f242d30750c23c5ea2571471822 not found: ID does not exist" Oct 30 17:57:18.595081 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.595059 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fqlng-zdr42"] Oct 30 17:57:18.601081 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:18.601063 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-fqlng-zdr42"] Oct 30 17:57:19.140876 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:19.140850 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a65a55c-27e9-4e39-a139-53ae907e0bfd" path="/var/lib/kubelet/pods/1a65a55c-27e9-4e39-a139-53ae907e0bfd/volumes" Oct 30 17:57:34.469879 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.469838 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-698f87866c-k4grf" podUID="78c06084-1bed-46c5-a576-f6129e21e062" containerName="registry" containerID="cri-o://05cb2273fb6cae9aed02024166258d2fa52277da54b5199cb2b56875c3ba74e0" gracePeriod=30 Oct 30 17:57:34.613933 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.613900 5014 generic.go:356] "Generic (PLEG): container finished" podID="78c06084-1bed-46c5-a576-f6129e21e062" containerID="05cb2273fb6cae9aed02024166258d2fa52277da54b5199cb2b56875c3ba74e0" exitCode=0 Oct 30 17:57:34.613933 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.613939 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698f87866c-k4grf" event={"ID":"78c06084-1bed-46c5-a576-f6129e21e062","Type":"ContainerDied","Data":"05cb2273fb6cae9aed02024166258d2fa52277da54b5199cb2b56875c3ba74e0"} Oct 30 17:57:34.759209 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.759178 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:57:34.856210 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.856182 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-kube-api-access-4k9gj\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-kube-api-access-4k9gj\") pod \"78c06084-1bed-46c5-a576-f6129e21e062\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " Oct 30 17:57:34.856382 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.856245 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-image-registry-private-configuration\") pod \"78c06084-1bed-46c5-a576-f6129e21e062\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " Oct 30 17:57:34.856382 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.856271 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-registry-certificates\") pod \"78c06084-1bed-46c5-a576-f6129e21e062\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " Oct 30 17:57:34.856382 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.856365 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\") pod \"78c06084-1bed-46c5-a576-f6129e21e062\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " Oct 30 17:57:34.856554 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.856405 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78c06084-1bed-46c5-a576-f6129e21e062-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c06084-1bed-46c5-a576-f6129e21e062-ca-trust-extracted\") pod \"78c06084-1bed-46c5-a576-f6129e21e062\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " Oct 30 17:57:34.856554 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.856488 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-installation-pull-secrets\") pod \"78c06084-1bed-46c5-a576-f6129e21e062\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " Oct 30 17:57:34.856554 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.856518 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-bound-sa-token\") pod \"78c06084-1bed-46c5-a576-f6129e21e062\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " Oct 30 17:57:34.856735 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.856555 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-trusted-ca\") pod \"78c06084-1bed-46c5-a576-f6129e21e062\" (UID: \"78c06084-1bed-46c5-a576-f6129e21e062\") " Oct 30 17:57:34.856794 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.856730 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-registry-certificates" pod "78c06084-1bed-46c5-a576-f6129e21e062" (UID: "78c06084-1bed-46c5-a576-f6129e21e062"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:34.856878 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.856860 5014 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-registry-certificates\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:34.857099 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.857076 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-trusted-ca" pod "78c06084-1bed-46c5-a576-f6129e21e062" (UID: "78c06084-1bed-46c5-a576-f6129e21e062"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:34.859064 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.859006 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls" pod "78c06084-1bed-46c5-a576-f6129e21e062" (UID: "78c06084-1bed-46c5-a576-f6129e21e062"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:34.859432 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.859399 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-installation-pull-secrets" pod "78c06084-1bed-46c5-a576-f6129e21e062" (UID: "78c06084-1bed-46c5-a576-f6129e21e062"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:34.859535 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.859450 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-kube-api-access-4k9gj" pod "78c06084-1bed-46c5-a576-f6129e21e062" (UID: "78c06084-1bed-46c5-a576-f6129e21e062"). InnerVolumeSpecName "kube-api-access-4k9gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:34.859535 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.859490 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-image-registry-private-configuration" pod "78c06084-1bed-46c5-a576-f6129e21e062" (UID: "78c06084-1bed-46c5-a576-f6129e21e062"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:34.860299 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.860273 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-bound-sa-token" pod "78c06084-1bed-46c5-a576-f6129e21e062" (UID: "78c06084-1bed-46c5-a576-f6129e21e062"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:34.864270 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.864233 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c06084-1bed-46c5-a576-f6129e21e062-ca-trust-extracted" pod "78c06084-1bed-46c5-a576-f6129e21e062" (UID: "78c06084-1bed-46c5-a576-f6129e21e062"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:34.957234 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.957209 5014 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-installation-pull-secrets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:34.957234 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.957231 5014 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-bound-sa-token\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:34.957234 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.957240 5014 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c06084-1bed-46c5-a576-f6129e21e062-trusted-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:34.957410 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.957248 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k9gj\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-kube-api-access-4k9gj\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:34.957410 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.957258 5014 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c06084-1bed-46c5-a576-f6129e21e062-image-registry-private-configuration\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:34.957410 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.957267 5014 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c06084-1bed-46c5-a576-f6129e21e062-registry-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:34.957410 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:34.957275 5014 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c06084-1bed-46c5-a576-f6129e21e062-ca-trust-extracted\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:57:35.622431 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:35.622398 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698f87866c-k4grf" event={"ID":"78c06084-1bed-46c5-a576-f6129e21e062","Type":"ContainerDied","Data":"c0be5292269a0a5c60fdcc459fe9621b41198e171416cea79f5fae32e7c1647e"} Oct 30 17:57:35.622926 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:35.622453 5014 scope.go:122] "RemoveContainer" containerID="05cb2273fb6cae9aed02024166258d2fa52277da54b5199cb2b56875c3ba74e0" Oct 30 17:57:35.622926 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:35.622451 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698f87866c-k4grf" Oct 30 17:57:35.688922 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:35.688897 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-698f87866c-k4grf"] Oct 30 17:57:35.700568 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:35.700536 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-698f87866c-k4grf"] Oct 30 17:57:37.141396 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:57:37.141362 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c06084-1bed-46c5-a576-f6129e21e062" path="/var/lib/kubelet/pods/78c06084-1bed-46c5-a576-f6129e21e062/volumes" Oct 30 17:57:43.331950 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:57:43.331910 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 17:58:01.729410 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:01.729378 5014 generic.go:356] "Generic (PLEG): container finished" podID="551adda7-a285-4f6e-9daf-b5b9f4410b48" containerID="86cd7b526512817d6867f8d107b196d3d02ecf33d2d87e9a9c97259d844a4616" exitCode=0 Oct 30 17:58:01.729802 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:01.729432 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" event={"ID":"551adda7-a285-4f6e-9daf-b5b9f4410b48","Type":"ContainerDied","Data":"86cd7b526512817d6867f8d107b196d3d02ecf33d2d87e9a9c97259d844a4616"} Oct 30 17:58:01.729802 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:01.729741 5014 scope.go:122] "RemoveContainer" containerID="86cd7b526512817d6867f8d107b196d3d02ecf33d2d87e9a9c97259d844a4616" Oct 30 17:58:02.735054 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:02.735018 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7xzrx" event={"ID":"551adda7-a285-4f6e-9daf-b5b9f4410b48","Type":"ContainerStarted","Data":"f05e21b511c0011b6f964b810902c76074ae615878ab8d8f46c46f35b2679c6b"} Oct 30 17:58:13.332977 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:58:13.332936 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 17:58:14.793007 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:14.792973 5014 generic.go:356] "Generic (PLEG): container finished" podID="2c6cb5c2-6a10-41ef-8d2e-aa337d19995f" containerID="c5a1c41d53d53371d6865b78ad1f11cec60dac6baf3736aa32e9199086b56b53" exitCode=0 Oct 30 17:58:14.793472 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:14.793047 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" event={"ID":"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f","Type":"ContainerDied","Data":"c5a1c41d53d53371d6865b78ad1f11cec60dac6baf3736aa32e9199086b56b53"} Oct 30 17:58:15.936003 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:15.935979 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:58:16.044778 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.044739 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-service-ca-bundle\") pod \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " Oct 30 17:58:16.044903 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.044799 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-trusted-ca-bundle\") pod \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " Oct 30 17:58:16.044903 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.044860 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-kube-api-access-g4r79\" (UniqueName: \"kubernetes.io/projected/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-kube-api-access-g4r79\") pod \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " Oct 30 17:58:16.044903 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.044894 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-archives-path\") pod \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\" (UID: \"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f\") " Oct 30 17:58:16.045097 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.045076 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-service-ca-bundle" pod "2c6cb5c2-6a10-41ef-8d2e-aa337d19995f" (UID: "2c6cb5c2-6a10-41ef-8d2e-aa337d19995f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:58:16.045239 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.045207 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-trusted-ca-bundle" pod "2c6cb5c2-6a10-41ef-8d2e-aa337d19995f" (UID: "2c6cb5c2-6a10-41ef-8d2e-aa337d19995f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:58:16.045343 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.045266 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-archives-path" pod "2c6cb5c2-6a10-41ef-8d2e-aa337d19995f" (UID: "2c6cb5c2-6a10-41ef-8d2e-aa337d19995f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:58:16.047088 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.047068 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-kube-api-access-g4r79" pod "2c6cb5c2-6a10-41ef-8d2e-aa337d19995f" (UID: "2c6cb5c2-6a10-41ef-8d2e-aa337d19995f"). InnerVolumeSpecName "kube-api-access-g4r79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:16.145967 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.145935 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4r79\" (UniqueName: \"kubernetes.io/projected/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-kube-api-access-g4r79\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:58:16.145967 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.145965 5014 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-archives-path\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:58:16.145967 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.145974 5014 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-service-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:58:16.146144 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.145983 5014 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c6cb5c2-6a10-41ef-8d2e-aa337d19995f-trusted-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 17:58:16.803217 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.803189 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" Oct 30 17:58:16.803412 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.803229 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bnqm4-s2tbr" event={"ID":"2c6cb5c2-6a10-41ef-8d2e-aa337d19995f","Type":"ContainerDied","Data":"804e11028a5d3846963acaea2bf6064759fb6a033303bfa29ee8e98b437e736d"} Oct 30 17:58:16.803412 ip-10-0-133-62 kubenswrapper[5014]: I1030 17:58:16.803275 5014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="804e11028a5d3846963acaea2bf6064759fb6a033303bfa29ee8e98b437e736d" Oct 30 17:58:43.333871 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:58:43.333834 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 17:59:13.332516 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:59:13.332480 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 17:59:43.332295 ip-10-0-133-62 kubenswrapper[5014]: E1030 17:59:43.332263 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:00:13.331872 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:00:13.331788 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:00:43.333624 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:00:43.333553 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:01:13.333359 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:01:13.333313 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:01:27.165079 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:01:27.165052 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:01:27.165681 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:01:27.165122 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:01:43.332309 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:01:43.332271 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:02:13.332614 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:02:13.332562 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:02:43.334718 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:02:43.334672 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:03:13.332879 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:03:13.332795 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:03:43.333509 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:03:43.333467 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:04:13.332819 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:04:13.332784 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:04:43.332689 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:04:43.332646 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:05:13.336442 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:05:13.336401 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:05:43.336085 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:05:43.336043 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:06:13.333012 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:06:13.332965 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:06:27.188015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:06:27.187983 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:06:27.189220 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:06:27.189201 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:06:43.333049 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:06:43.333013 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:07:13.332680 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:07:13.332645 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:07:43.332350 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:07:43.332315 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:08:13.332682 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:08:13.332645 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:08:43.334982 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:08:43.334941 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.209833 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8"] Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210475 5014 cpu_manager.go:414] "containerMap: removing container" podUID="1a65a55c-27e9-4e39-a139-53ae907e0bfd" containerName="insights-gathering" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210493 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a65a55c-27e9-4e39-a139-53ae907e0bfd" containerName="insights-gathering" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210518 5014 cpu_manager.go:414] "containerMap: removing container" podUID="78c06084-1bed-46c5-a576-f6129e21e062" containerName="registry" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210528 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c06084-1bed-46c5-a576-f6129e21e062" containerName="registry" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210576 5014 cpu_manager.go:414] "containerMap: removing container" podUID="2c6cb5c2-6a10-41ef-8d2e-aa337d19995f" containerName="insights-gathering" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210583 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c6cb5c2-6a10-41ef-8d2e-aa337d19995f" containerName="insights-gathering" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210631 5014 cpu_manager.go:414] "containerMap: removing container" podUID="1136549d-7491-452c-b396-f839668b5a19" containerName="registry" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210639 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1136549d-7491-452c-b396-f839668b5a19" containerName="registry" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210808 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1136549d-7491-452c-b396-f839668b5a19" containerName="registry" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210829 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a65a55c-27e9-4e39-a139-53ae907e0bfd" containerName="insights-gathering" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210838 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c06084-1bed-46c5-a576-f6129e21e062" containerName="registry" Oct 30 18:09:13.212532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.210855 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c6cb5c2-6a10-41ef-8d2e-aa337d19995f" containerName="insights-gathering" Oct 30 18:09:13.214503 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.214483 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8" Oct 30 18:09:13.230132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.230107 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-service-cert\"" Oct 30 18:09:13.238500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.238477 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd8f7e96-7ccb-46f8-ac00-0539a3503fdb-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8\" (UID: \"dd8f7e96-7ccb-46f8-ac00-0539a3503fdb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8" Oct 30 18:09:13.238589 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.238540 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd8f7e96-7ccb-46f8-ac00-0539a3503fdb-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8\" (UID: \"dd8f7e96-7ccb-46f8-ac00-0539a3503fdb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8" Oct 30 18:09:13.241482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.241463 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-dockercfg-tdgcq\"" Oct 30 18:09:13.242729 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.242709 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8"] Oct 30 18:09:13.331735 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:09:13.331705 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:09:13.339180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.339164 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd8f7e96-7ccb-46f8-ac00-0539a3503fdb-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8\" (UID: \"dd8f7e96-7ccb-46f8-ac00-0539a3503fdb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8" Oct 30 18:09:13.339266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.339211 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd8f7e96-7ccb-46f8-ac00-0539a3503fdb-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8\" (UID: \"dd8f7e96-7ccb-46f8-ac00-0539a3503fdb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8" Oct 30 18:09:13.341515 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.341495 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd8f7e96-7ccb-46f8-ac00-0539a3503fdb-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8\" (UID: \"dd8f7e96-7ccb-46f8-ac00-0539a3503fdb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8" Oct 30 18:09:13.341666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.341647 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd8f7e96-7ccb-46f8-ac00-0539a3503fdb-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8\" (UID: \"dd8f7e96-7ccb-46f8-ac00-0539a3503fdb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8" Oct 30 18:09:13.526406 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.526336 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8" Oct 30 18:09:13.856555 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.856520 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8"] Oct 30 18:09:13.862168 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:09:13.862146 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8f7e96_7ccb_46f8_ac00_0539a3503fdb.slice/crio-9f28d918adbf56a09ac49e4ab16277a2db3e2a5949816e59315e8f0d367e9aae WatchSource:0}: Error finding container 9f28d918adbf56a09ac49e4ab16277a2db3e2a5949816e59315e8f0d367e9aae: Status 404 returned error can't find the container with id 9f28d918adbf56a09ac49e4ab16277a2db3e2a5949816e59315e8f0d367e9aae Oct 30 18:09:13.865179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:13.865162 5014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:09:14.233750 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.233691 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6"] Oct 30 18:09:14.238166 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.238150 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.246358 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.246331 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvrjv\" (UniqueName: \"kubernetes.io/projected/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-kube-api-access-vvrjv\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.246462 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.246378 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-openshift-service-ca\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.246462 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.246399 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-apiservice-cert\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.246568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.246470 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-webhook-cert\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.264522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.264502 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"perses-operator-service-cert\"" Oct 30 18:09:14.268699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.268670 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"kube-root-ca.crt\"" Oct 30 18:09:14.276348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.276328 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:09:14.277472 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.277451 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6"] Oct 30 18:09:14.347288 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.347265 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-openshift-service-ca\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.347390 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.347295 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-apiservice-cert\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.347390 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.347317 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-webhook-cert\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.347390 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.347369 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvrjv\" (UniqueName: \"kubernetes.io/projected/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-kube-api-access-vvrjv\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.347963 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.347941 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-openshift-service-ca\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.349918 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.349899 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-webhook-cert\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.349918 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.349917 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-apiservice-cert\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.366268 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.366248 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"perses-operator-dockercfg-2qrpl\"" Oct 30 18:09:14.371701 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.371686 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvrjv\" (UniqueName: \"kubernetes.io/projected/d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4-kube-api-access-vvrjv\") pod \"perses-operator-6f4fb48785-spdr6\" (UID: \"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4\") " pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.478080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.478058 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8" event={"ID":"dd8f7e96-7ccb-46f8-ac00-0539a3503fdb","Type":"ContainerStarted","Data":"9f28d918adbf56a09ac49e4ab16277a2db3e2a5949816e59315e8f0d367e9aae"} Oct 30 18:09:14.550572 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.550523 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:14.895072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:14.895045 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6"] Oct 30 18:09:14.899475 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:09:14.899436 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f092c2_91e4_42f5_8dd5_5bb4de07f9e4.slice/crio-923d4a72a18b23fd1f48fa1fad4683792733fe2502ee1f7b9626a276e83acf02 WatchSource:0}: Error finding container 923d4a72a18b23fd1f48fa1fad4683792733fe2502ee1f7b9626a276e83acf02: Status 404 returned error can't find the container with id 923d4a72a18b23fd1f48fa1fad4683792733fe2502ee1f7b9626a276e83acf02 Oct 30 18:09:15.485090 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:15.485056 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" event={"ID":"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4","Type":"ContainerStarted","Data":"923d4a72a18b23fd1f48fa1fad4683792733fe2502ee1f7b9626a276e83acf02"} Oct 30 18:09:16.748747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.748714 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h"] Oct 30 18:09:16.753459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.753436 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.766817 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.766799 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-apiservice-cert\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.766875 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.766830 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tkd4\" (UniqueName: \"kubernetes.io/projected/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-kube-api-access-7tkd4\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.766939 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.766916 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-webhook-cert\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.767200 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.767176 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.771828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.771808 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h"] Oct 30 18:09:16.773212 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.773153 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-tempo-operator\"/\"kube-root-ca.crt\"" Oct 30 18:09:16.773212 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.773153 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"webhook-server-cert\"" Oct 30 18:09:16.773212 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.773200 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-tempo-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:09:16.773417 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.773376 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-controller-manager-dockercfg-fmnxx\"" Oct 30 18:09:16.781862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.781843 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-controller-service-cert\"" Oct 30 18:09:16.782152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.782138 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-metrics\"" Oct 30 18:09:16.868547 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.868522 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.868732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.868561 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-apiservice-cert\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.868732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.868580 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tkd4\" (UniqueName: \"kubernetes.io/projected/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-kube-api-access-7tkd4\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.868732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.868644 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-webhook-cert\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.871112 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.871091 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.871218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.871167 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-webhook-cert\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.871343 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.871324 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-apiservice-cert\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:16.891451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:16.891425 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tkd4\" (UniqueName: \"kubernetes.io/projected/45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c-kube-api-access-7tkd4\") pod \"tempo-operator-controller-fdfd9979b-tlv2h\" (UID: \"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c\") " pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:17.067676 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:17.067647 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:17.414965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:17.414899 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h"] Oct 30 18:09:17.419351 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:09:17.419320 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45657a3a_c6c6_4b3f_9600_e0dfdd3e9c5c.slice/crio-e778ecaae7526e074df40f43e92db6d2d570edccaeb1e39298ae65cf1dcef1ac WatchSource:0}: Error finding container e778ecaae7526e074df40f43e92db6d2d570edccaeb1e39298ae65cf1dcef1ac: Status 404 returned error can't find the container with id e778ecaae7526e074df40f43e92db6d2d570edccaeb1e39298ae65cf1dcef1ac Oct 30 18:09:17.495221 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:17.495194 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" event={"ID":"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c","Type":"ContainerStarted","Data":"e778ecaae7526e074df40f43e92db6d2d570edccaeb1e39298ae65cf1dcef1ac"} Oct 30 18:09:17.497017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:17.496996 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" event={"ID":"d7f092c2-91e4-42f5-8dd5-5bb4de07f9e4","Type":"ContainerStarted","Data":"c8c5c922b7e84db955e91bd86ee30a52bb46fae822c19b34e46f1ccca6f69556"} Oct 30 18:09:17.497140 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:17.497122 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:17.498695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:17.498673 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8" event={"ID":"dd8f7e96-7ccb-46f8-ac00-0539a3503fdb","Type":"ContainerStarted","Data":"168c4ddd6f89700b5c0cb0d3c31951f768aafbec3de3fdc33f442554a1a08cb8"} Oct 30 18:09:17.548814 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:17.548780 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" podStartSLOduration=1.734965999 podStartE2EDuration="3.548768781s" totalImagesPullingTime="1.813802782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:17.547512271 +0000 UTC m=+770.535852758" watchObservedRunningTime="2026-10-30 18:09:17.548768781 +0000 UTC m=+770.537109268" Oct 30 18:09:17.590074 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:17.590035 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-j6dw8" podStartSLOduration=1.961855857 podStartE2EDuration="4.590019703s" totalImagesPullingTime="2.628163846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:17.588266121 +0000 UTC m=+770.576606605" watchObservedRunningTime="2026-10-30 18:09:17.590019703 +0000 UTC m=+770.578360193" Oct 30 18:09:20.514181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:20.514090 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" event={"ID":"45657a3a-c6c6-4b3f-9600-e0dfdd3e9c5c","Type":"ContainerStarted","Data":"64a2166b112ba4e95c99431cce95f0a3bea8e24a4ade673d9c557c9f6ef1d9f7"} Oct 30 18:09:20.514181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:20.514146 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:20.582092 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:20.582018 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" podStartSLOduration=1.945822591 podStartE2EDuration="4.582002561s" totalImagesPullingTime="2.63617997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:20.571215814 +0000 UTC m=+773.559556301" watchObservedRunningTime="2026-10-30 18:09:20.582002561 +0000 UTC m=+773.570343073" Oct 30 18:09:28.505551 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:28.505519 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-cluster-observability-operator/perses-operator-6f4fb48785-spdr6" Oct 30 18:09:31.522116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:31.522082 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-tempo-operator/tempo-operator-controller-fdfd9979b-tlv2h" Oct 30 18:09:35.210740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.210696 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd"] Oct 30 18:09:35.214949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.214927 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:35.243932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.243904 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\"" Oct 30 18:09:35.244093 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.243950 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\"" Oct 30 18:09:35.244093 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.244033 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:09:35.244215 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.244114 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-85bvp\"" Oct 30 18:09:35.246737 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.246714 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd"] Oct 30 18:09:35.310142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.310119 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-apiservice-cert\") pod \"opentelemetry-operator-webhook-84fbc49bc7-5bsjd\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:35.310275 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.310157 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grd7\" (UniqueName: \"kubernetes.io/projected/9ccd290f-cce6-4153-8777-7c4dc0edaaab-kube-api-access-4grd7\") pod \"opentelemetry-operator-webhook-84fbc49bc7-5bsjd\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:35.310275 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.310188 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-webhook-cert\") pod \"opentelemetry-operator-webhook-84fbc49bc7-5bsjd\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:35.411579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.411540 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-apiservice-cert\") pod \"opentelemetry-operator-webhook-84fbc49bc7-5bsjd\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:35.411705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.411589 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grd7\" (UniqueName: \"kubernetes.io/projected/9ccd290f-cce6-4153-8777-7c4dc0edaaab-kube-api-access-4grd7\") pod \"opentelemetry-operator-webhook-84fbc49bc7-5bsjd\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:35.411705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.411635 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-webhook-cert\") pod \"opentelemetry-operator-webhook-84fbc49bc7-5bsjd\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:35.414069 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.414045 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-webhook-cert\") pod \"opentelemetry-operator-webhook-84fbc49bc7-5bsjd\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:35.414173 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.414083 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-apiservice-cert\") pod \"opentelemetry-operator-webhook-84fbc49bc7-5bsjd\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:35.434871 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.434853 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grd7\" (UniqueName: \"kubernetes.io/projected/9ccd290f-cce6-4153-8777-7c4dc0edaaab-kube-api-access-4grd7\") pod \"opentelemetry-operator-webhook-84fbc49bc7-5bsjd\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:35.526848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.526790 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:35.848481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:35.848449 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd"] Oct 30 18:09:35.853104 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:09:35.853069 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccd290f_cce6_4153_8777_7c4dc0edaaab.slice/crio-1f0fd2c2b99ce742212ab6dd90e3751a0cc63a2d9604aed3e82b20f5f7692eef WatchSource:0}: Error finding container 1f0fd2c2b99ce742212ab6dd90e3751a0cc63a2d9604aed3e82b20f5f7692eef: Status 404 returned error can't find the container with id 1f0fd2c2b99ce742212ab6dd90e3751a0cc63a2d9604aed3e82b20f5f7692eef Oct 30 18:09:36.596071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:36.596036 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" event={"ID":"9ccd290f-cce6-4153-8777-7c4dc0edaaab","Type":"ContainerStarted","Data":"1f0fd2c2b99ce742212ab6dd90e3751a0cc63a2d9604aed3e82b20f5f7692eef"} Oct 30 18:09:37.331384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:37.331355 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd"] Oct 30 18:09:37.601617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:37.601566 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" event={"ID":"9ccd290f-cce6-4153-8777-7c4dc0edaaab","Type":"ContainerStarted","Data":"6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0"} Oct 30 18:09:37.601972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:37.601653 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:37.668573 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:37.668534 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" podStartSLOduration=1.276932742 podStartE2EDuration="2.668520683s" totalImagesPullingTime="1.391587941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:37.656433778 +0000 UTC m=+790.644774276" watchObservedRunningTime="2026-10-30 18:09:37.668520683 +0000 UTC m=+790.656861170" Oct 30 18:09:38.607229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:38.607184 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" podUID="9ccd290f-cce6-4153-8777-7c4dc0edaaab" containerName="webhook" containerID="cri-o://6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0" gracePeriod=30 Oct 30 18:09:38.959373 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:38.959348 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:39.043856 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.043824 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccd290f-cce6-4153-8777-7c4dc0edaaab-kube-api-access-4grd7\" (UniqueName: \"kubernetes.io/projected/9ccd290f-cce6-4153-8777-7c4dc0edaaab-kube-api-access-4grd7\") pod \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " Oct 30 18:09:39.043996 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.043901 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-webhook-cert\") pod \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " Oct 30 18:09:39.043996 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.043952 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-apiservice-cert\") pod \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\" (UID: \"9ccd290f-cce6-4153-8777-7c4dc0edaaab\") " Oct 30 18:09:39.046156 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.046126 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccd290f-cce6-4153-8777-7c4dc0edaaab-kube-api-access-4grd7" pod "9ccd290f-cce6-4153-8777-7c4dc0edaaab" (UID: "9ccd290f-cce6-4153-8777-7c4dc0edaaab"). InnerVolumeSpecName "kube-api-access-4grd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:39.046156 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.046133 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-webhook-cert" pod "9ccd290f-cce6-4153-8777-7c4dc0edaaab" (UID: "9ccd290f-cce6-4153-8777-7c4dc0edaaab"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:39.046289 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.046199 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-apiservice-cert" pod "9ccd290f-cce6-4153-8777-7c4dc0edaaab" (UID: "9ccd290f-cce6-4153-8777-7c4dc0edaaab"). InnerVolumeSpecName "apiservice-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:39.145170 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.145117 5014 reconciler_common.go:299] "Volume detached for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-apiservice-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:09:39.145170 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.145137 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4grd7\" (UniqueName: \"kubernetes.io/projected/9ccd290f-cce6-4153-8777-7c4dc0edaaab-kube-api-access-4grd7\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:09:39.145170 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.145146 5014 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ccd290f-cce6-4153-8777-7c4dc0edaaab-webhook-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:09:39.612228 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.612194 5014 generic.go:356] "Generic (PLEG): container finished" podID="9ccd290f-cce6-4153-8777-7c4dc0edaaab" containerID="6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0" exitCode=0 Oct 30 18:09:39.612723 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.612235 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" Oct 30 18:09:39.612723 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.612249 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" event={"ID":"9ccd290f-cce6-4153-8777-7c4dc0edaaab","Type":"ContainerDied","Data":"6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0"} Oct 30 18:09:39.612723 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.612296 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd" event={"ID":"9ccd290f-cce6-4153-8777-7c4dc0edaaab","Type":"ContainerDied","Data":"1f0fd2c2b99ce742212ab6dd90e3751a0cc63a2d9604aed3e82b20f5f7692eef"} Oct 30 18:09:39.612723 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.612318 5014 scope.go:122] "RemoveContainer" containerID="6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0" Oct 30 18:09:39.621073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.621050 5014 scope.go:122] "RemoveContainer" containerID="6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0" Oct 30 18:09:39.621352 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:09:39.621330 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0\": container with ID starting with 6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0 not found: ID does not exist" containerID="6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0" Oct 30 18:09:39.621457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.621359 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0"} err="failed to get container status \"6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0\": rpc error: code = NotFound desc = could not find container \"6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0\": container with ID starting with 6ea59a4dfc7663c10ac43900a81886e390d18e3218e734b65bc445c781607bc0 not found: ID does not exist" Oct 30 18:09:39.669203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.669184 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd"] Oct 30 18:09:39.682199 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:39.682168 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-84fbc49bc7-5bsjd"] Oct 30 18:09:41.141471 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:09:41.141434 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ccd290f-cce6-4153-8777-7c4dc0edaaab" path="/var/lib/kubelet/pods/9ccd290f-cce6-4153-8777-7c4dc0edaaab/volumes" Oct 30 18:09:43.333076 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:09:43.333041 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:10:13.331505 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:10:13.331469 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:10:43.334286 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:10:43.334247 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:10:59.451814 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.451780 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:10:59.452435 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.452280 5014 cpu_manager.go:414] "containerMap: removing container" podUID="9ccd290f-cce6-4153-8777-7c4dc0edaaab" containerName="webhook" Oct 30 18:10:59.452435 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.452297 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccd290f-cce6-4153-8777-7c4dc0edaaab" containerName="webhook" Oct 30 18:10:59.452571 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.452454 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccd290f-cce6-4153-8777-7c4dc0edaaab" containerName="webhook" Oct 30 18:10:59.455811 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.455792 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.471583 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.471560 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:10:59.471747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.471730 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:10:59.483873 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.483849 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:10:59.485488 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.485444 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:10:59.485641 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.485614 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 18:10:59.485712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.485666 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:10:59.528388 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.528362 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b899509-26ca-4fce-94a4-e2b638d02ab6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.528520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.528400 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.528588 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.528523 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.528708 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.528688 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbpvn\" (UniqueName: \"kubernetes.io/projected/3b899509-26ca-4fce-94a4-e2b638d02ab6-kube-api-access-lbpvn\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.629985 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.629964 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbpvn\" (UniqueName: \"kubernetes.io/projected/3b899509-26ca-4fce-94a4-e2b638d02ab6-kube-api-access-lbpvn\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.630117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.629997 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b899509-26ca-4fce-94a4-e2b638d02ab6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.630117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.630017 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.630237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.630208 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.630670 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.630648 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b899509-26ca-4fce-94a4-e2b638d02ab6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.632555 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.632536 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.632732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.632713 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.652620 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.652569 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbpvn\" (UniqueName: \"kubernetes.io/projected/3b899509-26ca-4fce-94a4-e2b638d02ab6-kube-api-access-lbpvn\") pod \"data-science-collector-collector-0\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:10:59.769237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:10:59.769180 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:00.071138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:00.070780 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:11:00.074718 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:11:00.074694 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b899509_26ca_4fce_94a4_e2b638d02ab6.slice/crio-a4a18632d8f507402518111463cfbba8d2f3640442c0dcf903b0b870d2fb61bd WatchSource:0}: Error finding container a4a18632d8f507402518111463cfbba8d2f3640442c0dcf903b0b870d2fb61bd: Status 404 returned error can't find the container with id a4a18632d8f507402518111463cfbba8d2f3640442c0dcf903b0b870d2fb61bd Oct 30 18:11:00.966808 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:00.966776 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3b899509-26ca-4fce-94a4-e2b638d02ab6","Type":"ContainerStarted","Data":"a4a18632d8f507402518111463cfbba8d2f3640442c0dcf903b0b870d2fb61bd"} Oct 30 18:11:04.048926 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:04.048896 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:11:04.049218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:04.048953 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:11:04.990126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:04.990092 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3b899509-26ca-4fce-94a4-e2b638d02ab6","Type":"ContainerStarted","Data":"6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868"} Oct 30 18:11:05.033006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:05.032956 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.061795699 podStartE2EDuration="6.032943756s" totalImagesPullingTime="3.971148057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:05.031485298 +0000 UTC m=+878.019825797" watchObservedRunningTime="2026-10-30 18:11:05.032943756 +0000 UTC m=+878.021284319" Oct 30 18:11:13.336980 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:11:13.336874 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:11:27.214538 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:27.214506 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:11:27.216725 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:27.216699 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:11:35.748264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:35.748219 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:11:35.749229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:35.748859 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="3b899509-26ca-4fce-94a4-e2b638d02ab6" containerName="otc-container" containerID="cri-o://6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868" gracePeriod=30 Oct 30 18:11:36.001978 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.001929 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:36.085224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.084583 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b899509-26ca-4fce-94a4-e2b638d02ab6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b899509-26ca-4fce-94a4-e2b638d02ab6-otc-internal\") pod \"3b899509-26ca-4fce-94a4-e2b638d02ab6\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " Oct 30 18:11:36.085224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.084654 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-otlp-tls-certs\") pod \"3b899509-26ca-4fce-94a4-e2b638d02ab6\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " Oct 30 18:11:36.085224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.084695 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-monitoring-tls-certs\") pod \"3b899509-26ca-4fce-94a4-e2b638d02ab6\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " Oct 30 18:11:36.085224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.084749 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b899509-26ca-4fce-94a4-e2b638d02ab6-kube-api-access-lbpvn\" (UniqueName: \"kubernetes.io/projected/3b899509-26ca-4fce-94a4-e2b638d02ab6-kube-api-access-lbpvn\") pod \"3b899509-26ca-4fce-94a4-e2b638d02ab6\" (UID: \"3b899509-26ca-4fce-94a4-e2b638d02ab6\") " Oct 30 18:11:36.085724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.085329 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b899509-26ca-4fce-94a4-e2b638d02ab6-otc-internal" pod "3b899509-26ca-4fce-94a4-e2b638d02ab6" (UID: "3b899509-26ca-4fce-94a4-e2b638d02ab6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:36.087282 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.087251 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-otlp-tls-certs" pod "3b899509-26ca-4fce-94a4-e2b638d02ab6" (UID: "3b899509-26ca-4fce-94a4-e2b638d02ab6"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:36.087388 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.087344 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b899509-26ca-4fce-94a4-e2b638d02ab6-kube-api-access-lbpvn" pod "3b899509-26ca-4fce-94a4-e2b638d02ab6" (UID: "3b899509-26ca-4fce-94a4-e2b638d02ab6"). InnerVolumeSpecName "kube-api-access-lbpvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:36.087453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.087409 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-monitoring-tls-certs" pod "3b899509-26ca-4fce-94a4-e2b638d02ab6" (UID: "3b899509-26ca-4fce-94a4-e2b638d02ab6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:36.133114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.133082 5014 generic.go:356] "Generic (PLEG): container finished" podID="3b899509-26ca-4fce-94a4-e2b638d02ab6" containerID="6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868" exitCode=0 Oct 30 18:11:36.133239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.133124 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:36.133239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.133165 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3b899509-26ca-4fce-94a4-e2b638d02ab6","Type":"ContainerDied","Data":"6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868"} Oct 30 18:11:36.133239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.133232 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3b899509-26ca-4fce-94a4-e2b638d02ab6","Type":"ContainerDied","Data":"a4a18632d8f507402518111463cfbba8d2f3640442c0dcf903b0b870d2fb61bd"} Oct 30 18:11:36.133386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.133253 5014 scope.go:122] "RemoveContainer" containerID="6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868" Oct 30 18:11:36.143203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.143184 5014 scope.go:122] "RemoveContainer" containerID="6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868" Oct 30 18:11:36.143439 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:11:36.143415 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868\": container with ID starting with 6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868 not found: ID does not exist" containerID="6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868" Oct 30 18:11:36.143585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.143443 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868"} err="failed to get container status \"6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868\": rpc error: code = NotFound desc = could not find container \"6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868\": container with ID starting with 6efe4aff29969bb2f1abd80be8291182cd3fd9dcf54f5297e9ce8269b729b868 not found: ID does not exist" Oct 30 18:11:36.182386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.182366 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:11:36.185485 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.185465 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:11:36.185485 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.185486 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbpvn\" (UniqueName: \"kubernetes.io/projected/3b899509-26ca-4fce-94a4-e2b638d02ab6-kube-api-access-lbpvn\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:11:36.185643 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.185497 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b899509-26ca-4fce-94a4-e2b638d02ab6-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:11:36.185643 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.185506 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b899509-26ca-4fce-94a4-e2b638d02ab6-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:11:36.188159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:36.188140 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:11:37.141342 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:37.141313 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b899509-26ca-4fce-94a4-e2b638d02ab6" path="/var/lib/kubelet/pods/3b899509-26ca-4fce-94a4-e2b638d02ab6/volumes" Oct 30 18:11:41.505048 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.505017 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:11:41.505675 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.505482 5014 cpu_manager.go:414] "containerMap: removing container" podUID="3b899509-26ca-4fce-94a4-e2b638d02ab6" containerName="otc-container" Oct 30 18:11:41.505675 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.505501 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b899509-26ca-4fce-94a4-e2b638d02ab6" containerName="otc-container" Oct 30 18:11:41.505675 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.505635 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b899509-26ca-4fce-94a4-e2b638d02ab6" containerName="otc-container" Oct 30 18:11:41.508950 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.508931 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.523797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.523776 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:11:41.535476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.535453 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:11:41.535629 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.535604 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:11:41.535793 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.535520 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:11:41.535793 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.535525 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-b5f8018a\"" Oct 30 18:11:41.535910 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.535466 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:11:41.548368 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.548344 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:11:41.626910 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.626877 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wtfq\" (UniqueName: \"kubernetes.io/projected/665e47f0-5f74-4523-be75-73db859a7ed4-kube-api-access-7wtfq\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.627064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.626962 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.627064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.627003 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/665e47f0-5f74-4523-be75-73db859a7ed4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.627145 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.627112 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.627186 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.627165 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.727932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.727897 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.728104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.727943 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.728104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.727991 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtfq\" (UniqueName: \"kubernetes.io/projected/665e47f0-5f74-4523-be75-73db859a7ed4-kube-api-access-7wtfq\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.728104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.728017 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.728104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.728051 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/665e47f0-5f74-4523-be75-73db859a7ed4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.728277 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:11:41.728123 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:41.728277 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:11:41.728198 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert podName:665e47f0-5f74-4523-be75-73db859a7ed4 nodeName:}" failed. No retries permitted until 2026-10-30 18:11:42.228176627 +0000 UTC m=+915.216517114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "665e47f0-5f74-4523-be75-73db859a7ed4") : secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:41.728817 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.728739 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/665e47f0-5f74-4523-be75-73db859a7ed4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.730495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.730476 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.730582 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.730476 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:41.750660 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:41.750638 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtfq\" (UniqueName: \"kubernetes.io/projected/665e47f0-5f74-4523-be75-73db859a7ed4-kube-api-access-7wtfq\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:42.231772 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:42.231738 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:42.231923 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:11:42.231848 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:42.231923 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:11:42.231902 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert podName:665e47f0-5f74-4523-be75-73db859a7ed4 nodeName:}" failed. No retries permitted until 2026-10-30 18:11:43.231888985 +0000 UTC m=+916.220229455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "665e47f0-5f74-4523-be75-73db859a7ed4") : secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:43.239657 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:43.239621 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:43.240089 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:11:43.239755 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:43.240089 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:11:43.239826 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert podName:665e47f0-5f74-4523-be75-73db859a7ed4 nodeName:}" failed. No retries permitted until 2026-10-30 18:11:45.239808935 +0000 UTC m=+918.228149409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "665e47f0-5f74-4523-be75-73db859a7ed4") : secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:43.332078 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:11:43.332036 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:11:45.253609 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:45.253556 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:45.256260 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:45.256233 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:45.422508 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:45.422486 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:45.717818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:45.717787 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:11:45.721877 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:11:45.721839 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665e47f0_5f74_4523_be75_73db859a7ed4.slice/crio-458920488f4491cf476c228e72c207673349b3d52a59246b36541e09acbab0c1 WatchSource:0}: Error finding container 458920488f4491cf476c228e72c207673349b3d52a59246b36541e09acbab0c1: Status 404 returned error can't find the container with id 458920488f4491cf476c228e72c207673349b3d52a59246b36541e09acbab0c1 Oct 30 18:11:45.723924 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:45.723877 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:11:45.724026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:45.723964 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:11:46.178859 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:46.178827 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"665e47f0-5f74-4523-be75-73db859a7ed4","Type":"ContainerStarted","Data":"ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b"} Oct 30 18:11:46.179032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:46.178885 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"665e47f0-5f74-4523-be75-73db859a7ed4","Type":"ContainerStarted","Data":"458920488f4491cf476c228e72c207673349b3d52a59246b36541e09acbab0c1"} Oct 30 18:11:46.219958 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:46.219914 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=5.219901636 podStartE2EDuration="5.219901636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:46.218824873 +0000 UTC m=+919.207165470" watchObservedRunningTime="2026-10-30 18:11:46.219901636 +0000 UTC m=+919.208242191" Oct 30 18:11:56.898470 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:56.898426 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:11:56.898973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:56.898679 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="665e47f0-5f74-4523-be75-73db859a7ed4" containerName="otc-container" containerID="cri-o://ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b" gracePeriod=30 Oct 30 18:11:57.151341 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.151281 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:57.225295 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.225262 5014 generic.go:356] "Generic (PLEG): container finished" podID="665e47f0-5f74-4523-be75-73db859a7ed4" containerID="ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b" exitCode=0 Oct 30 18:11:57.225458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.225317 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:11:57.225458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.225322 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"665e47f0-5f74-4523-be75-73db859a7ed4","Type":"ContainerDied","Data":"ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b"} Oct 30 18:11:57.225458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.225372 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"665e47f0-5f74-4523-be75-73db859a7ed4","Type":"ContainerDied","Data":"458920488f4491cf476c228e72c207673349b3d52a59246b36541e09acbab0c1"} Oct 30 18:11:57.225458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.225401 5014 scope.go:122] "RemoveContainer" containerID="ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b" Oct 30 18:11:57.234935 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.234918 5014 scope.go:122] "RemoveContainer" containerID="ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b" Oct 30 18:11:57.235246 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:11:57.235224 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b\": container with ID starting with ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b not found: ID does not exist" containerID="ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b" Oct 30 18:11:57.235352 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.235250 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b"} err="failed to get container status \"ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b\": rpc error: code = NotFound desc = could not find container \"ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b\": container with ID starting with ab7f24671d677ee355b907d26ff2435d21f9ac20e496b5b5077773f87bfffa9b not found: ID does not exist" Oct 30 18:11:57.244354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.244335 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-monitoring-tls-certs\") pod \"665e47f0-5f74-4523-be75-73db859a7ed4\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " Oct 30 18:11:57.244470 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.244377 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-otlp-tls-certs\") pod \"665e47f0-5f74-4523-be75-73db859a7ed4\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " Oct 30 18:11:57.244470 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.244416 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665e47f0-5f74-4523-be75-73db859a7ed4-kube-api-access-7wtfq\" (UniqueName: \"kubernetes.io/projected/665e47f0-5f74-4523-be75-73db859a7ed4-kube-api-access-7wtfq\") pod \"665e47f0-5f74-4523-be75-73db859a7ed4\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " Oct 30 18:11:57.244587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.244474 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert\") pod \"665e47f0-5f74-4523-be75-73db859a7ed4\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " Oct 30 18:11:57.244587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.244517 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/665e47f0-5f74-4523-be75-73db859a7ed4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/665e47f0-5f74-4523-be75-73db859a7ed4-otc-internal\") pod \"665e47f0-5f74-4523-be75-73db859a7ed4\" (UID: \"665e47f0-5f74-4523-be75-73db859a7ed4\") " Oct 30 18:11:57.245328 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.245298 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/665e47f0-5f74-4523-be75-73db859a7ed4-otc-internal" pod "665e47f0-5f74-4523-be75-73db859a7ed4" (UID: "665e47f0-5f74-4523-be75-73db859a7ed4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:57.246938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.246911 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-otlp-tls-certs" pod "665e47f0-5f74-4523-be75-73db859a7ed4" (UID: "665e47f0-5f74-4523-be75-73db859a7ed4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:57.247181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.247157 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-monitoring-tls-certs" pod "665e47f0-5f74-4523-be75-73db859a7ed4" (UID: "665e47f0-5f74-4523-be75-73db859a7ed4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:57.247345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.247320 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert" pod "665e47f0-5f74-4523-be75-73db859a7ed4" (UID: "665e47f0-5f74-4523-be75-73db859a7ed4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:57.247439 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.247404 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665e47f0-5f74-4523-be75-73db859a7ed4-kube-api-access-7wtfq" pod "665e47f0-5f74-4523-be75-73db859a7ed4" (UID: "665e47f0-5f74-4523-be75-73db859a7ed4"). InnerVolumeSpecName "kube-api-access-7wtfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:57.345853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.345824 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:11:57.345853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.345850 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:11:57.345998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.345864 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wtfq\" (UniqueName: \"kubernetes.io/projected/665e47f0-5f74-4523-be75-73db859a7ed4-kube-api-access-7wtfq\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:11:57.345998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.345877 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/665e47f0-5f74-4523-be75-73db859a7ed4-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:11:57.345998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.345891 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/665e47f0-5f74-4523-be75-73db859a7ed4-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:11:57.567907 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.567879 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:11:57.575388 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:57.575368 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:11:59.142141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:11:59.142100 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665e47f0-5f74-4523-be75-73db859a7ed4" path="/var/lib/kubelet/pods/665e47f0-5f74-4523-be75-73db859a7ed4/volumes" Oct 30 18:12:00.989610 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:00.989562 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:00.990051 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:00.990034 5014 cpu_manager.go:414] "containerMap: removing container" podUID="665e47f0-5f74-4523-be75-73db859a7ed4" containerName="otc-container" Oct 30 18:12:00.990108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:00.990055 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665e47f0-5f74-4523-be75-73db859a7ed4" containerName="otc-container" Oct 30 18:12:00.990165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:00.990152 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="665e47f0-5f74-4523-be75-73db859a7ed4" containerName="otc-container" Oct 30 18:12:00.994432 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:00.994413 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.008658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.008636 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:12:01.008780 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.008753 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:12:01.021134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.021103 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:12:01.021134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.021105 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:12:01.021272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.021138 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-5hqkj\"" Oct 30 18:12:01.021272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.021182 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-c631c892\"" Oct 30 18:12:01.023119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.023093 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:01.172710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.172679 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.172855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.172730 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.172925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.172893 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.173098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.173076 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlvmq\" (UniqueName: \"kubernetes.io/projected/18c60e6d-3dfe-4ec6-b4b7-707a32786187-kube-api-access-jlvmq\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.274025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.273958 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.274025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.273998 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.274025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.274021 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.274230 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.274053 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlvmq\" (UniqueName: \"kubernetes.io/projected/18c60e6d-3dfe-4ec6-b4b7-707a32786187-kube-api-access-jlvmq\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.274569 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.274547 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.276638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.276589 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.276799 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.276779 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.296201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.296179 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlvmq\" (UniqueName: \"kubernetes.io/projected/18c60e6d-3dfe-4ec6-b4b7-707a32786187-kube-api-access-jlvmq\") pod \"data-science-collector-collector-0\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.309064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.309046 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:01.602806 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.602768 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:01.607251 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:12:01.607213 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c60e6d_3dfe_4ec6_b4b7_707a32786187.slice/crio-0c8091e918cdb615e5ea47e3e67dbc97efc922ef38cff9f06e499a5e71b3676a WatchSource:0}: Error finding container 0c8091e918cdb615e5ea47e3e67dbc97efc922ef38cff9f06e499a5e71b3676a: Status 404 returned error can't find the container with id 0c8091e918cdb615e5ea47e3e67dbc97efc922ef38cff9f06e499a5e71b3676a Oct 30 18:12:01.609215 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.609182 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:12:01.609319 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:01.609244 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:12:02.249261 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:02.249231 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"18c60e6d-3dfe-4ec6-b4b7-707a32786187","Type":"ContainerStarted","Data":"274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc"} Oct 30 18:12:02.249669 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:02.249274 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"18c60e6d-3dfe-4ec6-b4b7-707a32786187","Type":"ContainerStarted","Data":"0c8091e918cdb615e5ea47e3e67dbc97efc922ef38cff9f06e499a5e71b3676a"} Oct 30 18:12:02.291309 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:02.291244 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.291228993 podStartE2EDuration="2.291228993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:02.290083044 +0000 UTC m=+935.278423534" watchObservedRunningTime="2026-10-30 18:12:02.291228993 +0000 UTC m=+935.279569483" Oct 30 18:12:13.332041 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:12:13.331968 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:12:17.404626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.404580 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:17.405086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.404829 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="18c60e6d-3dfe-4ec6-b4b7-707a32786187" containerName="otc-container" containerID="cri-o://274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc" gracePeriod=30 Oct 30 18:12:17.647561 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.647534 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:17.694054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.693995 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otlp-tls-certs\") pod \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " Oct 30 18:12:17.694054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.694047 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c60e6d-3dfe-4ec6-b4b7-707a32786187-kube-api-access-jlvmq\" (UniqueName: \"kubernetes.io/projected/18c60e6d-3dfe-4ec6-b4b7-707a32786187-kube-api-access-jlvmq\") pod \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " Oct 30 18:12:17.694226 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.694148 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otc-internal\" (UniqueName: \"kubernetes.io/configmap/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otc-internal\") pod \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " Oct 30 18:12:17.694226 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.694187 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-monitoring-tls-certs\") pod \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\" (UID: \"18c60e6d-3dfe-4ec6-b4b7-707a32786187\") " Oct 30 18:12:17.694954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.694928 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otc-internal" pod "18c60e6d-3dfe-4ec6-b4b7-707a32786187" (UID: "18c60e6d-3dfe-4ec6-b4b7-707a32786187"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:17.699016 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.698983 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-monitoring-tls-certs" pod "18c60e6d-3dfe-4ec6-b4b7-707a32786187" (UID: "18c60e6d-3dfe-4ec6-b4b7-707a32786187"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:17.699111 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.699013 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c60e6d-3dfe-4ec6-b4b7-707a32786187-kube-api-access-jlvmq" pod "18c60e6d-3dfe-4ec6-b4b7-707a32786187" (UID: "18c60e6d-3dfe-4ec6-b4b7-707a32786187"). InnerVolumeSpecName "kube-api-access-jlvmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:17.699218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.699198 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otlp-tls-certs" pod "18c60e6d-3dfe-4ec6-b4b7-707a32786187" (UID: "18c60e6d-3dfe-4ec6-b4b7-707a32786187"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:17.794845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.794819 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:12:17.794845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.794843 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlvmq\" (UniqueName: \"kubernetes.io/projected/18c60e6d-3dfe-4ec6-b4b7-707a32786187-kube-api-access-jlvmq\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:12:17.794998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.794853 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18c60e6d-3dfe-4ec6-b4b7-707a32786187-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:12:17.794998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:17.794861 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18c60e6d-3dfe-4ec6-b4b7-707a32786187-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:12:18.318411 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:18.318378 5014 generic.go:356] "Generic (PLEG): container finished" podID="18c60e6d-3dfe-4ec6-b4b7-707a32786187" containerID="274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc" exitCode=0 Oct 30 18:12:18.318588 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:18.318427 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:18.318588 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:18.318465 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"18c60e6d-3dfe-4ec6-b4b7-707a32786187","Type":"ContainerDied","Data":"274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc"} Oct 30 18:12:18.318588 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:18.318525 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"18c60e6d-3dfe-4ec6-b4b7-707a32786187","Type":"ContainerDied","Data":"0c8091e918cdb615e5ea47e3e67dbc97efc922ef38cff9f06e499a5e71b3676a"} Oct 30 18:12:18.318588 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:18.318552 5014 scope.go:122] "RemoveContainer" containerID="274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc" Oct 30 18:12:18.327382 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:18.327362 5014 scope.go:122] "RemoveContainer" containerID="274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc" Oct 30 18:12:18.327655 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:12:18.327637 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc\": container with ID starting with 274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc not found: ID does not exist" containerID="274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc" Oct 30 18:12:18.327729 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:18.327664 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc"} err="failed to get container status \"274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc\": rpc error: code = NotFound desc = could not find container \"274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc\": container with ID starting with 274742a448b93dbfc8c5dc789eaee6ccdad88448ebe0c4a0f1799b900d43cbcc not found: ID does not exist" Oct 30 18:12:18.368316 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:18.368289 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:18.379644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:18.379619 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:19.141235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:19.141204 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c60e6d-3dfe-4ec6-b4b7-707a32786187" path="/var/lib/kubelet/pods/18c60e6d-3dfe-4ec6-b4b7-707a32786187/volumes" Oct 30 18:12:21.536484 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.536451 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:21.537201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.536949 5014 cpu_manager.go:414] "containerMap: removing container" podUID="18c60e6d-3dfe-4ec6-b4b7-707a32786187" containerName="otc-container" Oct 30 18:12:21.537201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.536969 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c60e6d-3dfe-4ec6-b4b7-707a32786187" containerName="otc-container" Oct 30 18:12:21.537201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.537104 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c60e6d-3dfe-4ec6-b4b7-707a32786187" containerName="otc-container" Oct 30 18:12:21.542066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.542043 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.556455 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.556430 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:12:21.556782 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.556764 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:12:21.568107 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.568087 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:12:21.568308 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.568217 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-4d267ad2\"" Oct 30 18:12:21.568380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.568347 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:12:21.568457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.568374 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:12:21.568632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.568562 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-kskgg\"" Oct 30 18:12:21.569843 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.569815 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:21.620522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.620497 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.620649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.620547 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.620649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.620566 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkv2\" (UniqueName: \"kubernetes.io/projected/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-kube-api-access-zhkv2\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.620724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.620699 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.620761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.620738 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.721802 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.721771 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.721802 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.721804 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkv2\" (UniqueName: \"kubernetes.io/projected/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-kube-api-access-zhkv2\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.722023 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.721827 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.722023 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.721856 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.722023 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.721889 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.722609 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.722554 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.724374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.724354 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.724488 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.724389 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.724548 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.724495 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.744190 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.744170 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkv2\" (UniqueName: \"kubernetes.io/projected/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-kube-api-access-zhkv2\") pod \"data-science-collector-collector-0\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:21.857641 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:21.857619 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:22.157411 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:22.157339 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:22.161368 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:12:22.161343 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda109f23d_d879_46d8_9a73_cb9cbd94eb0e.slice/crio-492c2e87e7c84cbcb42abbfc5c0e48597a22f6bc0cd9a1ad3234b52df7b04b61 WatchSource:0}: Error finding container 492c2e87e7c84cbcb42abbfc5c0e48597a22f6bc0cd9a1ad3234b52df7b04b61: Status 404 returned error can't find the container with id 492c2e87e7c84cbcb42abbfc5c0e48597a22f6bc0cd9a1ad3234b52df7b04b61 Oct 30 18:12:22.163255 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:22.163228 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:12:22.163326 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:22.163285 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:12:22.338138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:22.338103 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a109f23d-d879-46d8-9a73-cb9cbd94eb0e","Type":"ContainerStarted","Data":"c60f05ba867efa0bff0481c761b47587d51ebfd2d143bcf441232ce8337a2437"} Oct 30 18:12:22.338269 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:22.338172 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a109f23d-d879-46d8-9a73-cb9cbd94eb0e","Type":"ContainerStarted","Data":"492c2e87e7c84cbcb42abbfc5c0e48597a22f6bc0cd9a1ad3234b52df7b04b61"} Oct 30 18:12:22.379116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:22.379067 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.379055887 podStartE2EDuration="1.379055887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:22.378931029 +0000 UTC m=+955.367271516" watchObservedRunningTime="2026-10-30 18:12:22.379055887 +0000 UTC m=+955.367396374" Oct 30 18:12:37.178065 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.178029 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:37.178521 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.178283 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="a109f23d-d879-46d8-9a73-cb9cbd94eb0e" containerName="otc-container" containerID="cri-o://c60f05ba867efa0bff0481c761b47587d51ebfd2d143bcf441232ce8337a2437" gracePeriod=30 Oct 30 18:12:37.405779 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.405752 5014 generic.go:356] "Generic (PLEG): container finished" podID="a109f23d-d879-46d8-9a73-cb9cbd94eb0e" containerID="c60f05ba867efa0bff0481c761b47587d51ebfd2d143bcf441232ce8337a2437" exitCode=0 Oct 30 18:12:37.405901 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.405786 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a109f23d-d879-46d8-9a73-cb9cbd94eb0e","Type":"ContainerDied","Data":"c60f05ba867efa0bff0481c761b47587d51ebfd2d143bcf441232ce8337a2437"} Oct 30 18:12:37.422889 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.422869 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:37.532448 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.532385 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-data-science-collector-ta-client-cert\") pod \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " Oct 30 18:12:37.532448 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.532434 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-monitoring-tls-certs\") pod \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " Oct 30 18:12:37.532666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.532464 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-kube-api-access-zhkv2\" (UniqueName: \"kubernetes.io/projected/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-kube-api-access-zhkv2\") pod \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " Oct 30 18:12:37.532666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.532516 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otlp-tls-certs\") pod \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " Oct 30 18:12:37.532666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.532554 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otc-internal\") pod \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\" (UID: \"a109f23d-d879-46d8-9a73-cb9cbd94eb0e\") " Oct 30 18:12:37.532984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.532957 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otc-internal" pod "a109f23d-d879-46d8-9a73-cb9cbd94eb0e" (UID: "a109f23d-d879-46d8-9a73-cb9cbd94eb0e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:37.534584 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.534562 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-monitoring-tls-certs" pod "a109f23d-d879-46d8-9a73-cb9cbd94eb0e" (UID: "a109f23d-d879-46d8-9a73-cb9cbd94eb0e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:37.534704 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.534588 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-data-science-collector-ta-client-cert" pod "a109f23d-d879-46d8-9a73-cb9cbd94eb0e" (UID: "a109f23d-d879-46d8-9a73-cb9cbd94eb0e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:37.535167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.535149 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otlp-tls-certs" pod "a109f23d-d879-46d8-9a73-cb9cbd94eb0e" (UID: "a109f23d-d879-46d8-9a73-cb9cbd94eb0e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:37.535167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.535161 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-kube-api-access-zhkv2" pod "a109f23d-d879-46d8-9a73-cb9cbd94eb0e" (UID: "a109f23d-d879-46d8-9a73-cb9cbd94eb0e"). InnerVolumeSpecName "kube-api-access-zhkv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:37.633068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.633045 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:12:37.633068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.633066 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:12:37.633190 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.633075 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:12:37.633190 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.633084 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:12:37.633190 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:37.633093 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkv2\" (UniqueName: \"kubernetes.io/projected/a109f23d-d879-46d8-9a73-cb9cbd94eb0e-kube-api-access-zhkv2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:12:38.411652 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:38.411620 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a109f23d-d879-46d8-9a73-cb9cbd94eb0e","Type":"ContainerDied","Data":"492c2e87e7c84cbcb42abbfc5c0e48597a22f6bc0cd9a1ad3234b52df7b04b61"} Oct 30 18:12:38.411652 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:38.411636 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:38.412187 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:38.411675 5014 scope.go:122] "RemoveContainer" containerID="c60f05ba867efa0bff0481c761b47587d51ebfd2d143bcf441232ce8337a2437" Oct 30 18:12:38.460885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:38.460862 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:38.472459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:38.472439 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:39.141346 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:39.141312 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a109f23d-d879-46d8-9a73-cb9cbd94eb0e" path="/var/lib/kubelet/pods/a109f23d-d879-46d8-9a73-cb9cbd94eb0e/volumes" Oct 30 18:12:43.334955 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:12:43.334916 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:12:46.385438 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.385407 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq"] Oct 30 18:12:46.386018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.385794 5014 cpu_manager.go:414] "containerMap: removing container" podUID="a109f23d-d879-46d8-9a73-cb9cbd94eb0e" containerName="otc-container" Oct 30 18:12:46.386018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.385806 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a109f23d-d879-46d8-9a73-cb9cbd94eb0e" containerName="otc-container" Oct 30 18:12:46.386018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.385896 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="a109f23d-d879-46d8-9a73-cb9cbd94eb0e" containerName="otc-container" Oct 30 18:12:46.388695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.388672 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.410580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.410523 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq"] Oct 30 18:12:46.417310 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.417284 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:12:46.417432 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.417386 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:12:46.417759 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.417739 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:12:46.420328 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.420297 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:12:46.420451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.420329 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 18:12:46.420451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.420341 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-r2blm\"" Oct 30 18:12:46.420451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.420343 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 18:12:46.496109 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.496076 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.496256 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.496126 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.496256 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.496221 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgjk4\" (UniqueName: \"kubernetes.io/projected/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-api-access-rgjk4\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.496351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.496309 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.496351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.496342 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.596887 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.596857 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.597020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.596893 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.597020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.596915 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgjk4\" (UniqueName: \"kubernetes.io/projected/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-api-access-rgjk4\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.597020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.596968 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.597020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.596991 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.597020 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:12:46.597015 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 18:12:46.597266 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:12:46.597089 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-client-cert podName:2dc45bf9-d341-4619-876b-84e51bcedd38 nodeName:}" failed. No retries permitted until 2026-10-30 18:12:47.097072744 +0000 UTC m=+980.085413210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" (UID: "2dc45bf9-d341-4619-876b-84e51bcedd38") : secret "prometheus-operated-tls" not found Oct 30 18:12:46.597573 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.597554 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.599546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.599522 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.599644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.599532 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:46.619695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:46.619672 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgjk4\" (UniqueName: \"kubernetes.io/projected/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-api-access-rgjk4\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:47.101576 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.101545 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:47.103966 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.103939 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ntxjq\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:47.301490 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.301452 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:47.595993 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.595960 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq"] Oct 30 18:12:47.599789 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:12:47.599762 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc45bf9_d341_4619_876b_84e51bcedd38.slice/crio-db466b390840c281e705c4bda7a86ac66e2e83ac3ce292af5af64c019944e8cf WatchSource:0}: Error finding container db466b390840c281e705c4bda7a86ac66e2e83ac3ce292af5af64c019944e8cf: Status 404 returned error can't find the container with id db466b390840c281e705c4bda7a86ac66e2e83ac3ce292af5af64c019944e8cf Oct 30 18:12:47.939626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.939537 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:47.945881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.945859 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:47.972225 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.972160 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:12:47.972225 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.972186 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:12:47.972447 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.972244 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:12:47.972447 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.972345 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:12:47.972556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.972486 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:12:47.972661 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.972641 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wkhjr\"" Oct 30 18:12:47.975022 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:47.974886 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:48.009295 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.009266 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.009452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.009305 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.009452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.009425 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5d5v\" (UniqueName: \"kubernetes.io/projected/4eccebe0-6846-4aa9-8f46-93c986360370-kube-api-access-k5d5v\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.009568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.009478 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4eccebe0-6846-4aa9-8f46-93c986360370-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.009568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.009510 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.009716 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.009574 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.110531 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.110495 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.110732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.110574 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.110732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.110620 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.110732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.110664 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5d5v\" (UniqueName: \"kubernetes.io/projected/4eccebe0-6846-4aa9-8f46-93c986360370-kube-api-access-k5d5v\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.110732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.110687 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4eccebe0-6846-4aa9-8f46-93c986360370-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.110732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.110707 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.112023 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.111997 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4eccebe0-6846-4aa9-8f46-93c986360370-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.115338 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.115310 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.116260 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.116230 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.117513 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.117488 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.117863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.117839 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.135636 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.134972 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5d5v\" (UniqueName: \"kubernetes.io/projected/4eccebe0-6846-4aa9-8f46-93c986360370-kube-api-access-k5d5v\") pod \"data-science-collector-collector-0\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.263143 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.263075 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:12:48.459416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.459336 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" event={"ID":"2dc45bf9-d341-4619-876b-84e51bcedd38","Type":"ContainerStarted","Data":"db466b390840c281e705c4bda7a86ac66e2e83ac3ce292af5af64c019944e8cf"} Oct 30 18:12:48.570341 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.569984 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:12:48.575158 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:12:48.575120 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eccebe0_6846_4aa9_8f46_93c986360370.slice/crio-4b89612a3cccd378efd964b9d0e58a8d35b8e93843fc4007cd1c1b2a01199443 WatchSource:0}: Error finding container 4b89612a3cccd378efd964b9d0e58a8d35b8e93843fc4007cd1c1b2a01199443: Status 404 returned error can't find the container with id 4b89612a3cccd378efd964b9d0e58a8d35b8e93843fc4007cd1c1b2a01199443 Oct 30 18:12:48.577802 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.577767 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:12:48.577901 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:48.577841 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:12:49.465450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:49.465420 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4eccebe0-6846-4aa9-8f46-93c986360370","Type":"ContainerStarted","Data":"a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b"} Oct 30 18:12:49.465929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:49.465469 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4eccebe0-6846-4aa9-8f46-93c986360370","Type":"ContainerStarted","Data":"4b89612a3cccd378efd964b9d0e58a8d35b8e93843fc4007cd1c1b2a01199443"} Oct 30 18:12:49.509960 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:49.509626 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.50960745 podStartE2EDuration="2.50960745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:49.507283495 +0000 UTC m=+982.495624016" watchObservedRunningTime="2026-10-30 18:12:49.50960745 +0000 UTC m=+982.497947929" Oct 30 18:12:50.471354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:50.471285 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" event={"ID":"2dc45bf9-d341-4619-876b-84e51bcedd38","Type":"ContainerStarted","Data":"2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb"} Oct 30 18:12:50.511608 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:50.511543 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" podStartSLOduration=2.295237939 podStartE2EDuration="4.511530824s" totalImagesPullingTime="2.216292885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:50.511375857 +0000 UTC m=+983.499716343" watchObservedRunningTime="2026-10-30 18:12:50.511530824 +0000 UTC m=+983.499871310" Oct 30 18:12:51.475641 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:51.475586 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:55.194400 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:55.194374 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m"] Oct 30 18:12:55.198914 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:55.198896 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" Oct 30 18:12:55.218572 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:55.218546 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m"] Oct 30 18:12:55.229067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:55.229049 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-5mtd6\"" Oct 30 18:12:55.263280 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:55.263255 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mps6n\" (UniqueName: \"kubernetes.io/projected/513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1-kube-api-access-mps6n\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m\" (UID: \"513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" Oct 30 18:12:55.364135 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:55.364112 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mps6n\" (UniqueName: \"kubernetes.io/projected/513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1-kube-api-access-mps6n\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m\" (UID: \"513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" Oct 30 18:12:55.387417 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:55.387390 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mps6n\" (UniqueName: \"kubernetes.io/projected/513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1-kube-api-access-mps6n\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m\" (UID: \"513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" Oct 30 18:12:55.518187 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:55.518137 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" Oct 30 18:12:55.812512 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:55.812479 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m"] Oct 30 18:12:55.818280 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:12:55.818247 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod513732b1_c1cd_4e5a_a3db_9a62c8d9cdf1.slice/crio-a277332624b1ed7563c3b66dcff57cc052ff7023f5823e5dfc7f1bd255876461 WatchSource:0}: Error finding container a277332624b1ed7563c3b66dcff57cc052ff7023f5823e5dfc7f1bd255876461: Status 404 returned error can't find the container with id a277332624b1ed7563c3b66dcff57cc052ff7023f5823e5dfc7f1bd255876461 Oct 30 18:12:55.981679 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:55.981645 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:12:55.986951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:55.986933 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.002587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.002561 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:12:56.002701 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.002678 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:12:56.002816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.002792 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:12:56.002949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.002841 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-fjh62\"" Oct 30 18:12:56.002949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.002867 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:12:56.017180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.017155 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:12:56.070143 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.070086 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ht4\" (UniqueName: \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-kube-api-access-q6ht4\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.070249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.070142 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.070249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.070218 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.070319 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.070287 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.070354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.070317 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.070386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.070360 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.070386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.070382 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.171727 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.171699 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.171851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.171747 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.171851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.171776 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.171851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.171806 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.171851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.171837 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.172062 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.171871 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ht4\" (UniqueName: \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-kube-api-access-q6ht4\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.172062 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.171932 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.172369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.172309 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.174530 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.174499 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.174926 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.174900 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.175027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.175007 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.175122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.175102 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.175156 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.175140 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.197210 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.197185 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ht4\" (UniqueName: \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-kube-api-access-q6ht4\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.304633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.304583 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:12:56.498221 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.498195 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" event={"ID":"513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1","Type":"ContainerStarted","Data":"a277332624b1ed7563c3b66dcff57cc052ff7023f5823e5dfc7f1bd255876461"} Oct 30 18:12:56.633034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:56.632956 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:12:56.637049 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:12:56.637017 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4463cd_c3f6_4050_8ed4_574a8d733af3.slice/crio-00df2d712492fc5d2fbfba7cdf473d43cdf7409865f3389c7f0221ec0ab58f97 WatchSource:0}: Error finding container 00df2d712492fc5d2fbfba7cdf473d43cdf7409865f3389c7f0221ec0ab58f97: Status 404 returned error can't find the container with id 00df2d712492fc5d2fbfba7cdf473d43cdf7409865f3389c7f0221ec0ab58f97 Oct 30 18:12:57.482618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:57.482564 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:12:57.504870 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:57.504840 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6a4463cd-c3f6-4050-8ed4-574a8d733af3","Type":"ContainerStarted","Data":"00df2d712492fc5d2fbfba7cdf473d43cdf7409865f3389c7f0221ec0ab58f97"} Oct 30 18:12:58.511943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:58.511909 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6a4463cd-c3f6-4050-8ed4-574a8d733af3","Type":"ContainerStarted","Data":"547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170"} Oct 30 18:12:58.513430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:58.513407 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" event={"ID":"513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1","Type":"ContainerStarted","Data":"1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d"} Oct 30 18:12:58.607913 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:58.607869 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" podStartSLOduration=1.515422816 podStartE2EDuration="3.607856129s" totalImagesPullingTime="2.092433313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:58.606809538 +0000 UTC m=+991.595150025" watchObservedRunningTime="2026-10-30 18:12:58.607856129 +0000 UTC m=+991.596196617" Oct 30 18:12:59.141154 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.141118 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:12:59.144994 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.144978 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.159834 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.159806 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:12:59.159834 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.159823 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:12:59.160003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.159824 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:12:59.160003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.159855 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:12:59.160003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.159816 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:12:59.160003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.159881 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:12:59.160003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.159895 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:12:59.160003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.159909 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-tdqmt\"" Oct 30 18:12:59.171556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.171540 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:12:59.173217 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.173195 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:12:59.298207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298182 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.298357 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298224 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.298357 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298297 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.298357 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298336 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.298460 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298447 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01093ac4-bfb1-418e-9158-ce09cf82c600-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.298494 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298475 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.298531 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298499 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.298587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298574 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.298654 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298615 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.298654 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298648 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.298751 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298674 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.298751 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.298719 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s95mg\" (UniqueName: \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-kube-api-access-s95mg\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.399791 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.399725 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.399791 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.399757 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.399791 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.399774 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s95mg\" (UniqueName: \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-kube-api-access-s95mg\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.399832 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.399886 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.399926 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.399959 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.400006 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01093ac4-bfb1-418e-9158-ce09cf82c600-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.400041 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.400076 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.400121 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.400146 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400542 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.400514 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.400557 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.400983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.400956 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.403314 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.403285 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.403471 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.403431 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01093ac4-bfb1-418e-9158-ce09cf82c600-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.403471 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.403462 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.403806 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.403777 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.403898 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.403777 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.403980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.403960 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.404037 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.403973 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.404037 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.404014 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:12:59.422358 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:12:59.422337 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s95mg\" (UniqueName: \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-kube-api-access-s95mg\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:02.534641 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:02.534586 5014 generic.go:356] "Generic (PLEG): container finished" podID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerID="547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170" exitCode=0 Oct 30 18:13:02.535019 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:02.534658 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6a4463cd-c3f6-4050-8ed4-574a8d733af3","Type":"ContainerDied","Data":"547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170"} Oct 30 18:13:04.546668 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:04.546634 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6a4463cd-c3f6-4050-8ed4-574a8d733af3","Type":"ContainerStarted","Data":"69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4"} Oct 30 18:13:05.553836 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:05.553801 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6a4463cd-c3f6-4050-8ed4-574a8d733af3","Type":"ContainerStarted","Data":"6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46"} Oct 30 18:13:05.554303 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:05.553897 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:13:05.623958 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:05.623914 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=4.500919509 podStartE2EDuration="10.623902978s" totalImagesPullingTime="2.935281232s" totalInitContainerRuntime="3.187702237s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:05.61164963 +0000 UTC m=+998.599990116" watchObservedRunningTime="2026-10-30 18:13:05.623902978 +0000 UTC m=+998.612243465" Oct 30 18:13:06.658758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:06.658725 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:06.687151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:06.687126 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:06.759531 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:06.759508 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:06.759713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:06.759694 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:06.773749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:06.773724 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") DevicePath \"csi-3ece98a829a66f7cef6e6b4403002493a48d56f442e1673d82fff8fda237a009\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:06.959867 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:06.959838 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3d0cf4484736f6d40d16a5331fa2b97b9730899f1120a88b3f09811f40fc91e0/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:06.984408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:06.984386 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:07.263864 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:07.263789 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:07.604057 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:07.604026 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:13:07.607654 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:13:07.607628 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01093ac4_bfb1_418e_9158_ce09cf82c600.slice/crio-751a16b5e826a6556824c7af3771909112ababeebd8c7aa223c01684dc7fca9a WatchSource:0}: Error finding container 751a16b5e826a6556824c7af3771909112ababeebd8c7aa223c01684dc7fca9a: Status 404 returned error can't find the container with id 751a16b5e826a6556824c7af3771909112ababeebd8c7aa223c01684dc7fca9a Oct 30 18:13:08.571669 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:08.571633 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"01093ac4-bfb1-418e-9158-ce09cf82c600","Type":"ContainerStarted","Data":"9923096c604091aec7fca69b77011e75a9d90d3674da3529a08ecc427d00a31a"} Oct 30 18:13:08.572049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:08.571700 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"01093ac4-bfb1-418e-9158-ce09cf82c600","Type":"ContainerStarted","Data":"751a16b5e826a6556824c7af3771909112ababeebd8c7aa223c01684dc7fca9a"} Oct 30 18:13:11.559245 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:11.559221 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:13:11.587500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:11.587476 5014 generic.go:356] "Generic (PLEG): container finished" podID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerID="9923096c604091aec7fca69b77011e75a9d90d3674da3529a08ecc427d00a31a" exitCode=0 Oct 30 18:13:11.587637 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:11.587550 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"01093ac4-bfb1-418e-9158-ce09cf82c600","Type":"ContainerDied","Data":"9923096c604091aec7fca69b77011e75a9d90d3674da3529a08ecc427d00a31a"} Oct 30 18:13:13.336947 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:13:13.336910 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:13:15.611617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:15.611564 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"01093ac4-bfb1-418e-9158-ce09cf82c600","Type":"ContainerStarted","Data":"69d798223401ccf446aab228c2792f0bdddadb5d098b3f05514695baf923696d"} Oct 30 18:13:15.611985 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:15.611666 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"01093ac4-bfb1-418e-9158-ce09cf82c600","Type":"ContainerStarted","Data":"cfc41730ad8db477b7b7d9c5610d1b4c9ecc04cca3de498b58f39a613c6cd5a5"} Oct 30 18:13:15.611985 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:15.611701 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"01093ac4-bfb1-418e-9158-ce09cf82c600","Type":"ContainerStarted","Data":"cd39eaea997320d06b6150582d98ac70e0d3b7c0b5fcfc23b2857893e7511dbd"} Oct 30 18:13:15.697889 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:15.697795 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=13.192538773 podStartE2EDuration="19.697783543s" totalImagesPullingTime="3.318947623s" totalInitContainerRuntime="3.186297147s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:15.685693142 +0000 UTC m=+1008.674033629" watchObservedRunningTime="2026-10-30 18:13:15.697783543 +0000 UTC m=+1008.686124029" Oct 30 18:13:17.263954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:17.263927 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:22.265034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:22.265000 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:22.270692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:22.270671 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:22.648374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:22.648348 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:26.651425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:26.651386 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq"] Oct 30 18:13:26.651965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:26.651608 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" podUID="2dc45bf9-d341-4619-876b-84e51bcedd38" containerName="kube-rbac-proxy" containerID="cri-o://2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb" gracePeriod=30 Oct 30 18:13:26.957104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:26.957077 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:13:27.030980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.030941 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-tls\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-tls\") pod \"2dc45bf9-d341-4619-876b-84e51bcedd38\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " Oct 30 18:13:27.031164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.031007 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-client-cert\") pod \"2dc45bf9-d341-4619-876b-84e51bcedd38\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " Oct 30 18:13:27.031164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.031052 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-rbac-proxy-config\") pod \"2dc45bf9-d341-4619-876b-84e51bcedd38\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " Oct 30 18:13:27.031164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.031091 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-ca\") pod \"2dc45bf9-d341-4619-876b-84e51bcedd38\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " Oct 30 18:13:27.031164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.031141 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-api-access-rgjk4\" (UniqueName: \"kubernetes.io/projected/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-api-access-rgjk4\") pod \"2dc45bf9-d341-4619-876b-84e51bcedd38\" (UID: \"2dc45bf9-d341-4619-876b-84e51bcedd38\") " Oct 30 18:13:27.031547 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.031514 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-ca" pod "2dc45bf9-d341-4619-876b-84e51bcedd38" (UID: "2dc45bf9-d341-4619-876b-84e51bcedd38"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:27.033413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.033388 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-tls" pod "2dc45bf9-d341-4619-876b-84e51bcedd38" (UID: "2dc45bf9-d341-4619-876b-84e51bcedd38"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.033706 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.033678 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-api-access-rgjk4" pod "2dc45bf9-d341-4619-876b-84e51bcedd38" (UID: "2dc45bf9-d341-4619-876b-84e51bcedd38"). InnerVolumeSpecName "kube-api-access-rgjk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:27.033959 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.033933 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-rbac-proxy-config" pod "2dc45bf9-d341-4619-876b-84e51bcedd38" (UID: "2dc45bf9-d341-4619-876b-84e51bcedd38"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.034142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.034123 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-client-cert" pod "2dc45bf9-d341-4619-876b-84e51bcedd38" (UID: "2dc45bf9-d341-4619-876b-84e51bcedd38"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.072646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.072610 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m"] Oct 30 18:13:27.072852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.072818 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" podUID="513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1" containerName="thanos-querier" containerID="cri-o://1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d" gracePeriod=30 Oct 30 18:13:27.132621 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.132581 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.132621 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.132621 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.132812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.132631 5014 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-rbac-proxy-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.132812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.132640 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2dc45bf9-d341-4619-876b-84e51bcedd38-prometheus-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.132812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.132648 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgjk4\" (UniqueName: \"kubernetes.io/projected/2dc45bf9-d341-4619-876b-84e51bcedd38-kube-api-access-rgjk4\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.219034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.219004 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:13:27.219916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.219275 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="alertmanager" containerID="cri-o://69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4" gracePeriod=120 Oct 30 18:13:27.219916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.219379 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="config-reloader" containerID="cri-o://6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46" gracePeriod=120 Oct 30 18:13:27.248720 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.248679 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:13:27.248974 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.248928 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="prometheus" containerID="cri-o://cd39eaea997320d06b6150582d98ac70e0d3b7c0b5fcfc23b2857893e7511dbd" gracePeriod=600 Oct 30 18:13:27.249106 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.249087 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="thanos-sidecar" containerID="cri-o://69d798223401ccf446aab228c2792f0bdddadb5d098b3f05514695baf923696d" gracePeriod=600 Oct 30 18:13:27.249187 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.249170 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="config-reloader" containerID="cri-o://cfc41730ad8db477b7b7d9c5610d1b4c9ecc04cca3de498b58f39a613c6cd5a5" gracePeriod=600 Oct 30 18:13:27.378381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.378115 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" Oct 30 18:13:27.434957 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.434877 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1-kube-api-access-mps6n\" (UniqueName: \"kubernetes.io/projected/513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1-kube-api-access-mps6n\") pod \"513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1\" (UID: \"513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1\") " Oct 30 18:13:27.436965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.436938 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1-kube-api-access-mps6n" pod "513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1" (UID: "513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1"). InnerVolumeSpecName "kube-api-access-mps6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:27.535546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.535511 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mps6n\" (UniqueName: \"kubernetes.io/projected/513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1-kube-api-access-mps6n\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.645025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.644992 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="prometheus" probeResult="failure" output="Get \"https://10.132.0.44:9090/-/ready\": dial tcp 10.132.0.44:9090: connect: connection refused" Oct 30 18:13:27.669708 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.669680 5014 generic.go:356] "Generic (PLEG): container finished" podID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerID="6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46" exitCode=0 Oct 30 18:13:27.670082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.669758 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6a4463cd-c3f6-4050-8ed4-574a8d733af3","Type":"ContainerDied","Data":"6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46"} Oct 30 18:13:27.671006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.670984 5014 generic.go:356] "Generic (PLEG): container finished" podID="513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1" containerID="1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d" exitCode=0 Oct 30 18:13:27.671071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.671030 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" Oct 30 18:13:27.671071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.671052 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" event={"ID":"513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1","Type":"ContainerDied","Data":"1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d"} Oct 30 18:13:27.671149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.671088 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m" event={"ID":"513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1","Type":"ContainerDied","Data":"a277332624b1ed7563c3b66dcff57cc052ff7023f5823e5dfc7f1bd255876461"} Oct 30 18:13:27.671149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.671105 5014 scope.go:122] "RemoveContainer" containerID="1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d" Oct 30 18:13:27.672385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.672328 5014 generic.go:356] "Generic (PLEG): container finished" podID="2dc45bf9-d341-4619-876b-84e51bcedd38" containerID="2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb" exitCode=0 Oct 30 18:13:27.672385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.672377 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" Oct 30 18:13:27.672515 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.672379 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" event={"ID":"2dc45bf9-d341-4619-876b-84e51bcedd38","Type":"ContainerDied","Data":"2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb"} Oct 30 18:13:27.672515 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.672430 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq" event={"ID":"2dc45bf9-d341-4619-876b-84e51bcedd38","Type":"ContainerDied","Data":"db466b390840c281e705c4bda7a86ac66e2e83ac3ce292af5af64c019944e8cf"} Oct 30 18:13:27.675568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.675550 5014 generic.go:356] "Generic (PLEG): container finished" podID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerID="69d798223401ccf446aab228c2792f0bdddadb5d098b3f05514695baf923696d" exitCode=0 Oct 30 18:13:27.675568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.675566 5014 generic.go:356] "Generic (PLEG): container finished" podID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerID="cfc41730ad8db477b7b7d9c5610d1b4c9ecc04cca3de498b58f39a613c6cd5a5" exitCode=0 Oct 30 18:13:27.675724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.675574 5014 generic.go:356] "Generic (PLEG): container finished" podID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerID="cd39eaea997320d06b6150582d98ac70e0d3b7c0b5fcfc23b2857893e7511dbd" exitCode=0 Oct 30 18:13:27.675724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.675588 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"01093ac4-bfb1-418e-9158-ce09cf82c600","Type":"ContainerDied","Data":"69d798223401ccf446aab228c2792f0bdddadb5d098b3f05514695baf923696d"} Oct 30 18:13:27.675724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.675655 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"01093ac4-bfb1-418e-9158-ce09cf82c600","Type":"ContainerDied","Data":"cfc41730ad8db477b7b7d9c5610d1b4c9ecc04cca3de498b58f39a613c6cd5a5"} Oct 30 18:13:27.675724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.675682 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"01093ac4-bfb1-418e-9158-ce09cf82c600","Type":"ContainerDied","Data":"cd39eaea997320d06b6150582d98ac70e0d3b7c0b5fcfc23b2857893e7511dbd"} Oct 30 18:13:27.680464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.680450 5014 scope.go:122] "RemoveContainer" containerID="1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d" Oct 30 18:13:27.680767 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:13:27.680741 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d\": container with ID starting with 1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d not found: ID does not exist" containerID="1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d" Oct 30 18:13:27.680843 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.680770 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d"} err="failed to get container status \"1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d\": rpc error: code = NotFound desc = could not find container \"1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d\": container with ID starting with 1fe34749d64449fe04478581d52f4a3f858724ecd44f6fd13fa544cf0ec4f51d not found: ID does not exist" Oct 30 18:13:27.680843 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.680785 5014 scope.go:122] "RemoveContainer" containerID="2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb" Oct 30 18:13:27.688979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.688963 5014 scope.go:122] "RemoveContainer" containerID="2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb" Oct 30 18:13:27.689246 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:13:27.689225 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb\": container with ID starting with 2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb not found: ID does not exist" containerID="2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb" Oct 30 18:13:27.689282 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.689253 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb"} err="failed to get container status \"2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb\": rpc error: code = NotFound desc = could not find container \"2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb\": container with ID starting with 2b8156eae4e8cb39c159514f9550582b2ef037301895aef7df72f9c605b93fbb not found: ID does not exist" Oct 30 18:13:27.720303 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.720276 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq"] Oct 30 18:13:27.724998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.724979 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ntxjq"] Oct 30 18:13:27.759348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.759326 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m"] Oct 30 18:13:27.765474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.765455 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ww86m"] Oct 30 18:13:27.894920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.894889 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:13:27.895134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:27.895110 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="4eccebe0-6846-4aa9-8f46-93c986360370" containerName="otc-container" containerID="cri-o://a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b" gracePeriod=30 Oct 30 18:13:28.048019 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.047777 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:28.139963 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.139930 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-tls-assets\" (UniqueName: \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-tls-assets\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.139982 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01093ac4-bfb1-418e-9158-ce09cf82c600-config-out\" (UniqueName: \"kubernetes.io/empty-dir/01093ac4-bfb1-418e-9158-ce09cf82c600-config-out\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140024 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140065 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-kube-api-access-s95mg\" (UniqueName: \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-kube-api-access-s95mg\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140117 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140175 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140226 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-thanos-prometheus-http-client-file\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140253 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-config\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-config\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140290 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140337 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140456 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140560 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.140692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140623 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-secret-prometheus-web-tls-ca\") pod \"01093ac4-bfb1-418e-9158-ce09cf82c600\" (UID: \"01093ac4-bfb1-418e-9158-ce09cf82c600\") " Oct 30 18:13:28.141105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.140987 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-0" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:28.141610 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.141562 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-1" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:28.141993 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.141964 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-2" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:28.144663 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.144624 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-config" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.144864 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.144837 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-tls-assets" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:28.144951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.144922 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01093ac4-bfb1-418e-9158-ce09cf82c600-config-out" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:28.145161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.145131 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-thanos-prometheus-http-client-file" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.145259 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.145223 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-kube-api-access-s95mg" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "kube-api-access-s95mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:28.145501 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.145473 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.145564 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.145504 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.146199 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.146179 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-secret-prometheus-web-tls-ca" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.149123 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.149104 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:13:28.160803 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.160776 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config" pod "01093ac4-bfb1-418e-9158-ce09cf82c600" (UID: "01093ac4-bfb1-418e-9158-ce09cf82c600"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.165577 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.165560 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:28.241301 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241276 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-otlp-tls-certs\") pod \"4eccebe0-6846-4aa9-8f46-93c986360370\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " Oct 30 18:13:28.241433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241322 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-monitoring-tls-certs\") pod \"4eccebe0-6846-4aa9-8f46-93c986360370\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " Oct 30 18:13:28.241433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241361 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eccebe0-6846-4aa9-8f46-93c986360370-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4eccebe0-6846-4aa9-8f46-93c986360370-otc-internal\") pod \"4eccebe0-6846-4aa9-8f46-93c986360370\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " Oct 30 18:13:28.241433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241392 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-tls-certs\") pod \"4eccebe0-6846-4aa9-8f46-93c986360370\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " Oct 30 18:13:28.241433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241410 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eccebe0-6846-4aa9-8f46-93c986360370-kube-api-access-k5d5v\" (UniqueName: \"kubernetes.io/projected/4eccebe0-6846-4aa9-8f46-93c986360370-kube-api-access-k5d5v\") pod \"4eccebe0-6846-4aa9-8f46-93c986360370\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " Oct 30 18:13:28.241433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241428 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-data-science-collector-ta-client-cert\") pod \"4eccebe0-6846-4aa9-8f46-93c986360370\" (UID: \"4eccebe0-6846-4aa9-8f46-93c986360370\") " Oct 30 18:13:28.241816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241793 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.241816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241805 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eccebe0-6846-4aa9-8f46-93c986360370-otc-internal" pod "4eccebe0-6846-4aa9-8f46-93c986360370" (UID: "4eccebe0-6846-4aa9-8f46-93c986360370"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:28.242014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241820 5014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.242014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241837 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.242014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241853 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.242014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241869 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.242014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241907 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:13:28.242014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241925 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01093ac4-bfb1-418e-9158-ce09cf82c600-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.242014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241939 5014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.242014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241953 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.242014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241968 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01093ac4-bfb1-418e-9158-ce09cf82c600-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.242014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.241986 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.242014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.242005 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s95mg\" (UniqueName: \"kubernetes.io/projected/01093ac4-bfb1-418e-9158-ce09cf82c600-kube-api-access-s95mg\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.242576 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.242023 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/01093ac4-bfb1-418e-9158-ce09cf82c600-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.244067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.244039 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-tls-certs" pod "4eccebe0-6846-4aa9-8f46-93c986360370" (UID: "4eccebe0-6846-4aa9-8f46-93c986360370"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.244252 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.244230 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-otlp-tls-certs" pod "4eccebe0-6846-4aa9-8f46-93c986360370" (UID: "4eccebe0-6846-4aa9-8f46-93c986360370"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.244378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.244354 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eccebe0-6846-4aa9-8f46-93c986360370-kube-api-access-k5d5v" pod "4eccebe0-6846-4aa9-8f46-93c986360370" (UID: "4eccebe0-6846-4aa9-8f46-93c986360370"). InnerVolumeSpecName "kube-api-access-k5d5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:28.244977 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.244955 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-data-science-collector-ta-client-cert" pod "4eccebe0-6846-4aa9-8f46-93c986360370" (UID: "4eccebe0-6846-4aa9-8f46-93c986360370"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.245222 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.245199 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-monitoring-tls-certs" pod "4eccebe0-6846-4aa9-8f46-93c986360370" (UID: "4eccebe0-6846-4aa9-8f46-93c986360370"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.256041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.256024 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:13:28.342413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.342383 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-3ece98a829a66f7cef6e6b4403002493a48d56f442e1673d82fff8fda237a009\"" Oct 30 18:13:28.342413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.342409 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.342579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.342420 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.342579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.342429 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4eccebe0-6846-4aa9-8f46-93c986360370-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.342579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.342437 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.342579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.342445 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5d5v\" (UniqueName: \"kubernetes.io/projected/4eccebe0-6846-4aa9-8f46-93c986360370-kube-api-access-k5d5v\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.342579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.342453 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4eccebe0-6846-4aa9-8f46-93c986360370-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.682267 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.682186 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:13:28.682267 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.682201 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"01093ac4-bfb1-418e-9158-ce09cf82c600","Type":"ContainerDied","Data":"751a16b5e826a6556824c7af3771909112ababeebd8c7aa223c01684dc7fca9a"} Oct 30 18:13:28.682809 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.682313 5014 scope.go:122] "RemoveContainer" containerID="69d798223401ccf446aab228c2792f0bdddadb5d098b3f05514695baf923696d" Oct 30 18:13:28.684933 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.684894 5014 generic.go:356] "Generic (PLEG): container finished" podID="4eccebe0-6846-4aa9-8f46-93c986360370" containerID="a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b" exitCode=0 Oct 30 18:13:28.685056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.684940 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:28.685056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.684994 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4eccebe0-6846-4aa9-8f46-93c986360370","Type":"ContainerDied","Data":"a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b"} Oct 30 18:13:28.685056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.685033 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4eccebe0-6846-4aa9-8f46-93c986360370","Type":"ContainerDied","Data":"4b89612a3cccd378efd964b9d0e58a8d35b8e93843fc4007cd1c1b2a01199443"} Oct 30 18:13:28.695049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.694844 5014 scope.go:122] "RemoveContainer" containerID="cfc41730ad8db477b7b7d9c5610d1b4c9ecc04cca3de498b58f39a613c6cd5a5" Oct 30 18:13:28.712233 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.712216 5014 scope.go:122] "RemoveContainer" containerID="cd39eaea997320d06b6150582d98ac70e0d3b7c0b5fcfc23b2857893e7511dbd" Oct 30 18:13:28.720449 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.720429 5014 scope.go:122] "RemoveContainer" containerID="9923096c604091aec7fca69b77011e75a9d90d3674da3529a08ecc427d00a31a" Oct 30 18:13:28.728175 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.728160 5014 scope.go:122] "RemoveContainer" containerID="a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b" Oct 30 18:13:28.735321 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.735303 5014 scope.go:122] "RemoveContainer" containerID="a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b" Oct 30 18:13:28.735685 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:13:28.735664 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b\": container with ID starting with a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b not found: ID does not exist" containerID="a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b" Oct 30 18:13:28.735781 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.735696 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b"} err="failed to get container status \"a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b\": rpc error: code = NotFound desc = could not find container \"a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b\": container with ID starting with a00f4b556afcb409efcd1438d31e3d9d31223dfc3db12e81f3ae100dd9117c3b not found: ID does not exist" Oct 30 18:13:28.747482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.747458 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:13:28.759488 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.759464 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:13:28.784445 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.784424 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:13:28.792203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:28.792177 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:13:29.141471 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:29.141441 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" path="/var/lib/kubelet/pods/01093ac4-bfb1-418e-9158-ce09cf82c600/volumes" Oct 30 18:13:29.141950 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:29.141937 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc45bf9-d341-4619-876b-84e51bcedd38" path="/var/lib/kubelet/pods/2dc45bf9-d341-4619-876b-84e51bcedd38/volumes" Oct 30 18:13:29.142536 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:29.142522 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eccebe0-6846-4aa9-8f46-93c986360370" path="/var/lib/kubelet/pods/4eccebe0-6846-4aa9-8f46-93c986360370/volumes" Oct 30 18:13:29.142851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:29.142839 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1" path="/var/lib/kubelet/pods/513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1/volumes" Oct 30 18:13:31.559082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:31.559040 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.43:9093/-/ready\": dial tcp 10.132.0.43:9093: connect: connection refused" Oct 30 18:13:32.631346 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.631280 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:13:32.632091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.631965 5014 cpu_manager.go:414] "containerMap: removing container" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="init-config-reloader" Oct 30 18:13:32.632091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.631987 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="init-config-reloader" Oct 30 18:13:32.632091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632011 5014 cpu_manager.go:414] "containerMap: removing container" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="prometheus" Oct 30 18:13:32.632091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632018 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="prometheus" Oct 30 18:13:32.632091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632031 5014 cpu_manager.go:414] "containerMap: removing container" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="thanos-sidecar" Oct 30 18:13:32.632091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632038 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="thanos-sidecar" Oct 30 18:13:32.632091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632049 5014 cpu_manager.go:414] "containerMap: removing container" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="config-reloader" Oct 30 18:13:32.632091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632057 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="config-reloader" Oct 30 18:13:32.632091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632071 5014 cpu_manager.go:414] "containerMap: removing container" podUID="2dc45bf9-d341-4619-876b-84e51bcedd38" containerName="kube-rbac-proxy" Oct 30 18:13:32.632091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632077 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc45bf9-d341-4619-876b-84e51bcedd38" containerName="kube-rbac-proxy" Oct 30 18:13:32.632954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632104 5014 cpu_manager.go:414] "containerMap: removing container" podUID="4eccebe0-6846-4aa9-8f46-93c986360370" containerName="otc-container" Oct 30 18:13:32.632954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632111 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eccebe0-6846-4aa9-8f46-93c986360370" containerName="otc-container" Oct 30 18:13:32.632954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632124 5014 cpu_manager.go:414] "containerMap: removing container" podUID="513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1" containerName="thanos-querier" Oct 30 18:13:32.632954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632134 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1" containerName="thanos-querier" Oct 30 18:13:32.632954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632262 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eccebe0-6846-4aa9-8f46-93c986360370" containerName="otc-container" Oct 30 18:13:32.632954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632276 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="513732b1-c1cd-4e5a-a3db-9a62c8d9cdf1" containerName="thanos-querier" Oct 30 18:13:32.632954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632291 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="prometheus" Oct 30 18:13:32.632954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632301 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="thanos-sidecar" Oct 30 18:13:32.632954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632310 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="01093ac4-bfb1-418e-9158-ce09cf82c600" containerName="config-reloader" Oct 30 18:13:32.632954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.632328 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc45bf9-d341-4619-876b-84e51bcedd38" containerName="kube-rbac-proxy" Oct 30 18:13:32.637443 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.637422 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.654102 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.654080 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:13:32.665938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.665915 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-zr47k\"" Oct 30 18:13:32.668658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.668640 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:13:32.674048 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.674023 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.674118 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.674076 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.674215 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.674198 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxn5\" (UniqueName: \"kubernetes.io/projected/0eb21f1c-dac5-4fae-abf4-f269ce1db248-kube-api-access-7jxn5\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.674292 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.674276 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.677663 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.677643 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 18:13:32.679830 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.679808 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:13:32.775315 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.775288 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.775430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.775326 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.775430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.775368 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxn5\" (UniqueName: \"kubernetes.io/projected/0eb21f1c-dac5-4fae-abf4-f269ce1db248-kube-api-access-7jxn5\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.775430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.775393 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.775994 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.775977 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.777938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.777913 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.778072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.778049 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.798205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.798176 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxn5\" (UniqueName: \"kubernetes.io/projected/0eb21f1c-dac5-4fae-abf4-f269ce1db248-kube-api-access-7jxn5\") pod \"data-science-collector-collector-1\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:32.950433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:32.950377 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:33.246199 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:33.246120 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:13:33.252460 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:33.252427 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:13:33.252547 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:33.252483 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:13:33.712940 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:33.712885 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0eb21f1c-dac5-4fae-abf4-f269ce1db248","Type":"ContainerStarted","Data":"7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66"} Oct 30 18:13:33.713333 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:33.712948 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0eb21f1c-dac5-4fae-abf4-f269ce1db248","Type":"ContainerStarted","Data":"686ef5459374c8708179b3e3149d687bd3bf7ead13b0bae19de7bbb1913dbbce"} Oct 30 18:13:33.755330 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:33.755280 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.755267054 podStartE2EDuration="1.755267054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:33.753802407 +0000 UTC m=+1026.742142894" watchObservedRunningTime="2026-10-30 18:13:33.755267054 +0000 UTC m=+1026.743607542" Oct 30 18:13:36.559629 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:36.559571 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.43:9093/-/ready\": dial tcp 10.132.0.43:9093: connect: connection refused" Oct 30 18:13:37.354084 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:13:37.354047 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4463cd_c3f6_4050_8ed4_574a8d733af3.slice/crio-69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4463cd_c3f6_4050_8ed4_574a8d733af3.slice/crio-conmon-69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:13:37.485246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.485198 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:13:37.614329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.614302 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-alertmanager-data-science-monitoringstack-db\") pod \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " Oct 30 18:13:37.614768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.614332 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-tls-assets\" (UniqueName: \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-tls-assets\") pod \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " Oct 30 18:13:37.614768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.614373 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-web-config\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-web-config\") pod \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " Oct 30 18:13:37.614768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.614397 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-cluster-tls-config\") pod \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " Oct 30 18:13:37.614768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.614449 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-out\") pod \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " Oct 30 18:13:37.614768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.614471 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-kube-api-access-q6ht4\" (UniqueName: \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-kube-api-access-q6ht4\") pod \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " Oct 30 18:13:37.614768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.614499 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-volume\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-volume\") pod \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\" (UID: \"6a4463cd-c3f6-4050-8ed4-574a8d733af3\") " Oct 30 18:13:37.614768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.614703 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-alertmanager-data-science-monitoringstack-db" pod "6a4463cd-c3f6-4050-8ed4-574a8d733af3" (UID: "6a4463cd-c3f6-4050-8ed4-574a8d733af3"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:37.617006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.616975 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-out" pod "6a4463cd-c3f6-4050-8ed4-574a8d733af3" (UID: "6a4463cd-c3f6-4050-8ed4-574a8d733af3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:37.617221 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.617200 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-volume" pod "6a4463cd-c3f6-4050-8ed4-574a8d733af3" (UID: "6a4463cd-c3f6-4050-8ed4-574a8d733af3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:37.617292 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.617202 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-kube-api-access-q6ht4" pod "6a4463cd-c3f6-4050-8ed4-574a8d733af3" (UID: "6a4463cd-c3f6-4050-8ed4-574a8d733af3"). InnerVolumeSpecName "kube-api-access-q6ht4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:37.618700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.618680 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-tls-assets" pod "6a4463cd-c3f6-4050-8ed4-574a8d733af3" (UID: "6a4463cd-c3f6-4050-8ed4-574a8d733af3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:37.622178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.622152 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-cluster-tls-config" pod "6a4463cd-c3f6-4050-8ed4-574a8d733af3" (UID: "6a4463cd-c3f6-4050-8ed4-574a8d733af3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:37.629412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.629360 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-web-config" pod "6a4463cd-c3f6-4050-8ed4-574a8d733af3" (UID: "6a4463cd-c3f6-4050-8ed4-574a8d733af3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:37.715807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.715785 5014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:37.715807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.715808 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:37.715927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.715817 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:37.715927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.715825 5014 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-cluster-tls-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:37.715927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.715834 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:37.715927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.715843 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6ht4\" (UniqueName: \"kubernetes.io/projected/6a4463cd-c3f6-4050-8ed4-574a8d733af3-kube-api-access-q6ht4\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:37.715927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.715851 5014 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a4463cd-c3f6-4050-8ed4-574a8d733af3-config-volume\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:13:37.733842 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.733816 5014 generic.go:356] "Generic (PLEG): container finished" podID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerID="69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4" exitCode=0 Oct 30 18:13:37.733938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.733890 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:13:37.733938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.733905 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6a4463cd-c3f6-4050-8ed4-574a8d733af3","Type":"ContainerDied","Data":"69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4"} Oct 30 18:13:37.734007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.733963 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6a4463cd-c3f6-4050-8ed4-574a8d733af3","Type":"ContainerDied","Data":"00df2d712492fc5d2fbfba7cdf473d43cdf7409865f3389c7f0221ec0ab58f97"} Oct 30 18:13:37.734007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.733992 5014 scope.go:122] "RemoveContainer" containerID="6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46" Oct 30 18:13:37.742342 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.742321 5014 scope.go:122] "RemoveContainer" containerID="69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4" Oct 30 18:13:37.750305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.750290 5014 scope.go:122] "RemoveContainer" containerID="547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170" Oct 30 18:13:37.757447 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.757426 5014 scope.go:122] "RemoveContainer" containerID="6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46" Oct 30 18:13:37.757718 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:13:37.757699 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46\": container with ID starting with 6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46 not found: ID does not exist" containerID="6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46" Oct 30 18:13:37.757766 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.757729 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46"} err="failed to get container status \"6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46\": rpc error: code = NotFound desc = could not find container \"6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46\": container with ID starting with 6a87916c7354521e161ac78208843d9aa6964372edb96359a36e27d1b5039d46 not found: ID does not exist" Oct 30 18:13:37.757766 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.757748 5014 scope.go:122] "RemoveContainer" containerID="69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4" Oct 30 18:13:37.757983 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:13:37.757966 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4\": container with ID starting with 69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4 not found: ID does not exist" containerID="69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4" Oct 30 18:13:37.758049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.757996 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4"} err="failed to get container status \"69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4\": rpc error: code = NotFound desc = could not find container \"69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4\": container with ID starting with 69e13b31e4fb7891bccda58d096400efa5242aed306918c65d5511063c07c8c4 not found: ID does not exist" Oct 30 18:13:37.758049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.758018 5014 scope.go:122] "RemoveContainer" containerID="547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170" Oct 30 18:13:37.758260 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:13:37.758239 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170\": container with ID starting with 547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170 not found: ID does not exist" containerID="547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170" Oct 30 18:13:37.758312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.758265 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170"} err="failed to get container status \"547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170\": rpc error: code = NotFound desc = could not find container \"547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170\": container with ID starting with 547da3fc87a9040f76cb21811d19589d22ac064483044c7b1683e3cae9c95170 not found: ID does not exist" Oct 30 18:13:37.785396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.785377 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:13:37.790430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:37.790411 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:13:39.141464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:13:39.141429 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" path="/var/lib/kubelet/pods/6a4463cd-c3f6-4050-8ed4-574a8d733af3/volumes" Oct 30 18:13:43.333404 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:13:43.333364 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:14:13.332835 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:14:13.332794 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:14:22.220110 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.220074 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:14:22.221055 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.220354 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0eb21f1c-dac5-4fae-abf4-f269ce1db248" containerName="otc-container" containerID="cri-o://7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66" gracePeriod=30 Oct 30 18:14:22.484615 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.484535 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:22.635186 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.635157 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otc-internal\") pod \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " Oct 30 18:14:22.635336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.635193 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otlp-tls-certs\") pod \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " Oct 30 18:14:22.635336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.635226 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-monitoring-tls-certs\") pod \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " Oct 30 18:14:22.635336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.635248 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb21f1c-dac5-4fae-abf4-f269ce1db248-kube-api-access-7jxn5\" (UniqueName: \"kubernetes.io/projected/0eb21f1c-dac5-4fae-abf4-f269ce1db248-kube-api-access-7jxn5\") pod \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\" (UID: \"0eb21f1c-dac5-4fae-abf4-f269ce1db248\") " Oct 30 18:14:22.635575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.635537 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otc-internal" pod "0eb21f1c-dac5-4fae-abf4-f269ce1db248" (UID: "0eb21f1c-dac5-4fae-abf4-f269ce1db248"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:22.637296 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.637270 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otlp-tls-certs" pod "0eb21f1c-dac5-4fae-abf4-f269ce1db248" (UID: "0eb21f1c-dac5-4fae-abf4-f269ce1db248"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:22.637374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.637291 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-monitoring-tls-certs" pod "0eb21f1c-dac5-4fae-abf4-f269ce1db248" (UID: "0eb21f1c-dac5-4fae-abf4-f269ce1db248"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:22.637570 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.637545 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb21f1c-dac5-4fae-abf4-f269ce1db248-kube-api-access-7jxn5" pod "0eb21f1c-dac5-4fae-abf4-f269ce1db248" (UID: "0eb21f1c-dac5-4fae-abf4-f269ce1db248"). InnerVolumeSpecName "kube-api-access-7jxn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:22.736087 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.736006 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:14:22.736087 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.736039 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:14:22.736087 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.736053 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb21f1c-dac5-4fae-abf4-f269ce1db248-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:14:22.736087 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.736068 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jxn5\" (UniqueName: \"kubernetes.io/projected/0eb21f1c-dac5-4fae-abf4-f269ce1db248-kube-api-access-7jxn5\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:14:22.941222 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.941188 5014 generic.go:356] "Generic (PLEG): container finished" podID="0eb21f1c-dac5-4fae-abf4-f269ce1db248" containerID="7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66" exitCode=0 Oct 30 18:14:22.941393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.941241 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:22.941393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.941281 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0eb21f1c-dac5-4fae-abf4-f269ce1db248","Type":"ContainerDied","Data":"7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66"} Oct 30 18:14:22.941393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.941346 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0eb21f1c-dac5-4fae-abf4-f269ce1db248","Type":"ContainerDied","Data":"686ef5459374c8708179b3e3149d687bd3bf7ead13b0bae19de7bbb1913dbbce"} Oct 30 18:14:22.941393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.941369 5014 scope.go:122] "RemoveContainer" containerID="7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66" Oct 30 18:14:22.950031 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.950012 5014 scope.go:122] "RemoveContainer" containerID="7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66" Oct 30 18:14:22.950273 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:14:22.950254 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66\": container with ID starting with 7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66 not found: ID does not exist" containerID="7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66" Oct 30 18:14:22.950324 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.950280 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66"} err="failed to get container status \"7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66\": rpc error: code = NotFound desc = could not find container \"7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66\": container with ID starting with 7922d7bae65728a262c64b878db3f0ea7286aed4787821c4529f9ee583bcbe66 not found: ID does not exist" Oct 30 18:14:22.991319 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.991233 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:14:22.995828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:22.995808 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:14:23.141313 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:23.141289 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb21f1c-dac5-4fae-abf4-f269ce1db248" path="/var/lib/kubelet/pods/0eb21f1c-dac5-4fae-abf4-f269ce1db248/volumes" Oct 30 18:14:31.168190 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168161 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89"] Oct 30 18:14:31.168655 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168637 5014 cpu_manager.go:414] "containerMap: removing container" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="alertmanager" Oct 30 18:14:31.168722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168658 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="alertmanager" Oct 30 18:14:31.168722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168673 5014 cpu_manager.go:414] "containerMap: removing container" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="init-config-reloader" Oct 30 18:14:31.168722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168682 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="init-config-reloader" Oct 30 18:14:31.168722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168697 5014 cpu_manager.go:414] "containerMap: removing container" podUID="0eb21f1c-dac5-4fae-abf4-f269ce1db248" containerName="otc-container" Oct 30 18:14:31.168722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168705 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb21f1c-dac5-4fae-abf4-f269ce1db248" containerName="otc-container" Oct 30 18:14:31.168722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168714 5014 cpu_manager.go:414] "containerMap: removing container" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="config-reloader" Oct 30 18:14:31.168722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168721 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="config-reloader" Oct 30 18:14:31.168929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168821 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="alertmanager" Oct 30 18:14:31.168929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168834 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb21f1c-dac5-4fae-abf4-f269ce1db248" containerName="otc-container" Oct 30 18:14:31.168929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.168841 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4463cd-c3f6-4050-8ed4-574a8d733af3" containerName="config-reloader" Oct 30 18:14:31.173755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.173734 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.201148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.201122 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:14:31.201258 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.201190 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:14:31.201258 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.201245 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:14:31.201361 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.201341 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:14:31.202417 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.202400 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89"] Oct 30 18:14:31.204026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.204011 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-9jjcb\"" Oct 30 18:14:31.204284 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.204263 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 18:14:31.204284 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.204278 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 18:14:31.291517 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.291483 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.291668 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.291527 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.291668 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.291631 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4lkd\" (UniqueName: \"kubernetes.io/projected/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-api-access-t4lkd\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.291807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.291735 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.291807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.291782 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.392999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.392976 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.392999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.393000 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.393136 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.393027 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lkd\" (UniqueName: \"kubernetes.io/projected/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-api-access-t4lkd\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.393136 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.393046 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.393136 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:14:31.393132 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 18:14:31.393297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.393178 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.393297 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:14:31.393186 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-client-cert podName:28925fa7-e5d2-45c5-98ec-d0183301fd7f nodeName:}" failed. No retries permitted until 2026-10-30 18:14:31.893167941 +0000 UTC m=+1084.881508406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-z6v89" (UID: "28925fa7-e5d2-45c5-98ec-d0183301fd7f") : secret "prometheus-operated-tls" not found Oct 30 18:14:31.393788 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.393767 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.395529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.395514 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.395633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.395614 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.416145 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.416123 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lkd\" (UniqueName: \"kubernetes.io/projected/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-api-access-t4lkd\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.896887 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.896857 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:31.899287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:31.899259 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-z6v89\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:32.086767 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.086741 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:32.380951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.380912 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89"] Oct 30 18:14:32.386231 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:14:32.386206 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28925fa7_e5d2_45c5_98ec_d0183301fd7f.slice/crio-7a8c696be3887e8d7ed780e9e792c0687105ef29f4ca91ff1f9aaea1ddd16b12 WatchSource:0}: Error finding container 7a8c696be3887e8d7ed780e9e792c0687105ef29f4ca91ff1f9aaea1ddd16b12: Status 404 returned error can't find the container with id 7a8c696be3887e8d7ed780e9e792c0687105ef29f4ca91ff1f9aaea1ddd16b12 Oct 30 18:14:32.817790 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.817762 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:14:32.822820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.822795 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:32.841876 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.841847 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:14:32.851850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.851827 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:14:32.852007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.851846 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-mbgzb\"" Oct 30 18:14:32.852007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.851896 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:14:32.852007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.851953 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:14:32.852160 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.852007 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:14:32.852160 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.852048 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 30 18:14:32.906063 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.906039 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:32.906157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.906079 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:32.906157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.906103 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wl4\" (UniqueName: \"kubernetes.io/projected/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-kube-api-access-p8wl4\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:32.906244 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.906171 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:32.906244 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.906228 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:32.906313 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.906249 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:32.990937 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.990863 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" event={"ID":"28925fa7-e5d2-45c5-98ec-d0183301fd7f","Type":"ContainerStarted","Data":"9110306a3b1bee5e93e9be0b60ff41dd03cf8c8d7c98feee8b24094c5471307e"} Oct 30 18:14:32.990937 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.990910 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" event={"ID":"28925fa7-e5d2-45c5-98ec-d0183301fd7f","Type":"ContainerStarted","Data":"7a8c696be3887e8d7ed780e9e792c0687105ef29f4ca91ff1f9aaea1ddd16b12"} Oct 30 18:14:32.991181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:32.991015 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:33.007325 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.007300 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.007464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.007334 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wl4\" (UniqueName: \"kubernetes.io/projected/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-kube-api-access-p8wl4\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.007464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.007350 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.007464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.007381 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.007464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.007411 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.007699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.007474 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.008364 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.008337 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.010031 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.010003 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.010135 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.010120 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.010272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.010241 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.010574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.010549 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.032902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.032851 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" podStartSLOduration=2.032835828 podStartE2EDuration="2.032835828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:33.031062838 +0000 UTC m=+1086.019403326" watchObservedRunningTime="2026-10-30 18:14:33.032835828 +0000 UTC m=+1086.021176314" Oct 30 18:14:33.033252 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.033229 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wl4\" (UniqueName: \"kubernetes.io/projected/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-kube-api-access-p8wl4\") pod \"data-science-collector-collector-0\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.138463 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.138431 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:33.435559 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.435523 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:14:33.440649 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:14:33.440617 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f3c362_e4dc_4711_bfe3_90e3aa5f1d60.slice/crio-6ea84ab35c40e53044569aa6474fdc1d6a5c6b50047519ab67bbdc3c3dfb3988 WatchSource:0}: Error finding container 6ea84ab35c40e53044569aa6474fdc1d6a5c6b50047519ab67bbdc3c3dfb3988: Status 404 returned error can't find the container with id 6ea84ab35c40e53044569aa6474fdc1d6a5c6b50047519ab67bbdc3c3dfb3988 Oct 30 18:14:33.443035 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.443001 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:14:33.443117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.443075 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:14:33.998186 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.998150 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60","Type":"ContainerStarted","Data":"462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d"} Oct 30 18:14:33.998379 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:33.998199 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60","Type":"ContainerStarted","Data":"6ea84ab35c40e53044569aa6474fdc1d6a5c6b50047519ab67bbdc3c3dfb3988"} Oct 30 18:14:34.040811 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.040761 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.040744232 podStartE2EDuration="2.040744232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:34.039233867 +0000 UTC m=+1087.027574354" watchObservedRunningTime="2026-10-30 18:14:34.040744232 +0000 UTC m=+1087.029084720" Oct 30 18:14:34.255134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.254987 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:14:34.259215 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.259192 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.276365 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.276342 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:14:34.276502 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.276485 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:14:34.276581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.276529 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:14:34.276581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.276535 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:14:34.276734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.276716 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:14:34.276780 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.276723 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:14:34.276812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.276798 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:14:34.276812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.276806 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-28dh5\"" Oct 30 18:14:34.289902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.289878 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:14:34.418912 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.418881 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.419041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.418916 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.419041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.418935 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.419041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.418951 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.419144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.419076 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.419144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.419117 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.419144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.419142 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.419256 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.419186 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgln\" (UniqueName: \"kubernetes.io/projected/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-kube-api-access-szgln\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.519984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.519921 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.519984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.519956 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.519984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.519976 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgln\" (UniqueName: \"kubernetes.io/projected/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-kube-api-access-szgln\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.520511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.520013 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.520511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.520044 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.520511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.520068 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.520511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.520096 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.520511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.520173 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.520811 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.520766 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.520900 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.520874 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.521021 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.520912 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.521270 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.521252 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.522773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.522751 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.523072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.523057 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.523278 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.523257 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:34.543291 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:34.543269 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgln\" (UniqueName: \"kubernetes.io/projected/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-kube-api-access-szgln\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:38.548490 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:38.548456 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:38.576503 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:38.576477 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:38.649353 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:38.649329 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:38.649450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:38.649418 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:38.663432 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:38.663409 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:38.765973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:38.765949 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:38.792612 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:38.792575 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:39.000174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:39.000156 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:14:39.074818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:39.074793 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:39.404900 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:39.404874 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:14:39.408291 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:14:39.408261 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254b74cd_53ce_47bd_bf7d_a0d5c7b34289.slice/crio-149b30bdc73f840f14f7a227ae4865e7c90f04546614c51345738d27daae2362 WatchSource:0}: Error finding container 149b30bdc73f840f14f7a227ae4865e7c90f04546614c51345738d27daae2362: Status 404 returned error can't find the container with id 149b30bdc73f840f14f7a227ae4865e7c90f04546614c51345738d27daae2362 Oct 30 18:14:39.410165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:39.410148 5014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:14:40.025600 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:40.025571 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"254b74cd-53ce-47bd-bf7d-a0d5c7b34289","Type":"ContainerStarted","Data":"149b30bdc73f840f14f7a227ae4865e7c90f04546614c51345738d27daae2362"} Oct 30 18:14:41.023350 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.023315 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:14:41.028570 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.028539 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.038684 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.036275 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:14:41.043104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.043079 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.052621 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.050074 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:14:41.055749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.055721 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:14:41.092262 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.092238 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:14:41.092262 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.092251 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:14:41.092262 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.092251 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:14:41.092519 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.092343 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:14:41.092519 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.092343 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:14:41.092672 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.092540 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:14:41.092672 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.092559 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:14:41.092672 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.092642 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-5zvln\"" Oct 30 18:14:41.092824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.092700 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:14:41.092824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.092644 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:14:41.092824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.092729 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-l5rwl\"" Oct 30 18:14:41.108284 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.108266 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:14:41.108416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.108295 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:14:41.108506 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.108491 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:14:41.171281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171247 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.171392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171284 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.171392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171316 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.171482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171396 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.171482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171422 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.171482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171464 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.171646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171546 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.171646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171634 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.171741 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171668 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m55hq\" (UniqueName: \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-kube-api-access-m55hq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.171741 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171703 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.171830 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171778 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/324a7d04-26bd-4298-882d-1b77fd9532e4-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.171830 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171814 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.171911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171870 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.171961 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171906 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.171961 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.171933 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m7m2\" (UniqueName: \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-kube-api-access-4m7m2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.172031 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.172005 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.172076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.172046 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.172076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.172072 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.172180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.172095 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.273218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273185 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.273401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273235 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.273401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273271 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.273401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273312 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m55hq\" (UniqueName: \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-kube-api-access-m55hq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.273401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273343 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.273401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273370 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/324a7d04-26bd-4298-882d-1b77fd9532e4-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.273401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273391 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.273754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273425 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.273754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273460 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.273754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273487 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m7m2\" (UniqueName: \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-kube-api-access-4m7m2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.273754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273519 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.273754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273548 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.273754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273671 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.273754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273713 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.274002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273756 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.274002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273786 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.274002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273831 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.274002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273865 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.274002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.273890 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.274383 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.274361 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.274951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.274927 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.276677 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.276270 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.278719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.278681 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.279425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.278887 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/324a7d04-26bd-4298-882d-1b77fd9532e4-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.279425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.279003 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.279425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.279337 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.279730 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.279705 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.280344 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.280310 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.280467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.280449 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.281235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.281215 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.281531 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.281463 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.281747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.281725 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.281812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.281770 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.281904 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.281888 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.282015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.281995 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.283107 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.283091 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.298020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.297998 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m7m2\" (UniqueName: \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-kube-api-access-4m7m2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:41.298308 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.298288 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m55hq\" (UniqueName: \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-kube-api-access-m55hq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.367254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.367203 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:41.702030 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:41.701933 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:14:41.705627 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:14:41.705568 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18fc695_c6e6_489b_82c5_37da338a241e.slice/crio-190623d8d2862111d67af1ad8b538fec6a0e5597952c65a6a14a761e9cd8d066 WatchSource:0}: Error finding container 190623d8d2862111d67af1ad8b538fec6a0e5597952c65a6a14a761e9cd8d066: Status 404 returned error can't find the container with id 190623d8d2862111d67af1ad8b538fec6a0e5597952c65a6a14a761e9cd8d066 Oct 30 18:14:42.036910 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:42.036827 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b18fc695-c6e6-489b-82c5-37da338a241e","Type":"ContainerStarted","Data":"190623d8d2862111d67af1ad8b538fec6a0e5597952c65a6a14a761e9cd8d066"} Oct 30 18:14:42.038412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:42.038389 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"254b74cd-53ce-47bd-bf7d-a0d5c7b34289","Type":"ContainerStarted","Data":"f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7"} Oct 30 18:14:43.046443 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:43.045713 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b18fc695-c6e6-489b-82c5-37da338a241e","Type":"ContainerStarted","Data":"ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175"} Oct 30 18:14:43.337518 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:14:43.337486 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:14:45.056293 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:45.056259 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"254b74cd-53ce-47bd-bf7d-a0d5c7b34289","Type":"ContainerStarted","Data":"54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5"} Oct 30 18:14:46.063125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:46.063094 5014 generic.go:356] "Generic (PLEG): container finished" podID="b18fc695-c6e6-489b-82c5-37da338a241e" containerID="ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175" exitCode=0 Oct 30 18:14:46.063568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:46.063174 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b18fc695-c6e6-489b-82c5-37da338a241e","Type":"ContainerDied","Data":"ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175"} Oct 30 18:14:47.069960 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:47.069915 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b18fc695-c6e6-489b-82c5-37da338a241e","Type":"ContainerStarted","Data":"4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed"} Oct 30 18:14:47.070422 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:47.070022 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b18fc695-c6e6-489b-82c5-37da338a241e","Type":"ContainerStarted","Data":"16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2"} Oct 30 18:14:47.070422 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:47.070254 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:47.072144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:47.072121 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"254b74cd-53ce-47bd-bf7d-a0d5c7b34289","Type":"ContainerStarted","Data":"ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d"} Oct 30 18:14:47.072352 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:47.072332 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:47.072425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:47.072378 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:47.072425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:47.072403 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:47.074813 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:47.074797 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:47.081122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:47.081105 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:47.150249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:47.150210 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.763563932 podStartE2EDuration="7.150199819s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.386635887s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:47.149249343 +0000 UTC m=+1100.137589831" watchObservedRunningTime="2026-10-30 18:14:47.150199819 +0000 UTC m=+1100.138540305" Oct 30 18:14:47.214147 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:47.214100 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=6.199927198 podStartE2EDuration="13.214083837s" totalImagesPullingTime="7.014156639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:14:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:47.212435332 +0000 UTC m=+1100.200775818" watchObservedRunningTime="2026-10-30 18:14:47.214083837 +0000 UTC m=+1100.202424324" Oct 30 18:14:48.637840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:48.637811 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:48.665370 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:48.665349 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:48.738816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:48.738793 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:48.738946 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:48.738927 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:48.753063 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:48.753042 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:48.855782 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:48.855757 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/26c702426f1aeab2e9e6116e994cbd362b1230f690bb6d32002e36ee94fa3aec/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:48.882027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:48.882003 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:49.183811 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:49.183781 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:14:49.514742 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:49.514669 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:14:49.517652 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:14:49.517624 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324a7d04_26bd_4298_882d_1b77fd9532e4.slice/crio-398f266445aa21ea567c4d760ef52ed10e9f5fb0a134fb22d5d3aaebdbd51ddd WatchSource:0}: Error finding container 398f266445aa21ea567c4d760ef52ed10e9f5fb0a134fb22d5d3aaebdbd51ddd: Status 404 returned error can't find the container with id 398f266445aa21ea567c4d760ef52ed10e9f5fb0a134fb22d5d3aaebdbd51ddd Oct 30 18:14:50.087068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:50.087037 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"324a7d04-26bd-4298-882d-1b77fd9532e4","Type":"ContainerStarted","Data":"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b"} Oct 30 18:14:50.087380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:50.087112 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"324a7d04-26bd-4298-882d-1b77fd9532e4","Type":"ContainerStarted","Data":"398f266445aa21ea567c4d760ef52ed10e9f5fb0a134fb22d5d3aaebdbd51ddd"} Oct 30 18:14:53.079001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:53.078970 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:14:54.106646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:54.106589 5014 generic.go:356] "Generic (PLEG): container finished" podID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerID="353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b" exitCode=0 Oct 30 18:14:54.107030 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:54.106658 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"324a7d04-26bd-4298-882d-1b77fd9532e4","Type":"ContainerDied","Data":"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b"} Oct 30 18:14:55.114102 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:55.114071 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"324a7d04-26bd-4298-882d-1b77fd9532e4","Type":"ContainerStarted","Data":"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd"} Oct 30 18:14:55.114477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:55.114157 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"324a7d04-26bd-4298-882d-1b77fd9532e4","Type":"ContainerStarted","Data":"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc"} Oct 30 18:14:55.114477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:55.114183 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"324a7d04-26bd-4298-882d-1b77fd9532e4","Type":"ContainerStarted","Data":"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b"} Oct 30 18:14:55.188465 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:55.188413 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=11.803558738 podStartE2EDuration="15.188401139s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.384842401s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:55.187340375 +0000 UTC m=+1108.175680862" watchObservedRunningTime="2026-10-30 18:14:55.188401139 +0000 UTC m=+1108.176741695" Oct 30 18:14:59.083976 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:59.083936 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:14:59.184117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:14:59.184093 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:04.184451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:04.184408 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:04.189544 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:04.189525 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:05.164800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:05.164771 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:09.083042 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:09.082952 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:15:13.333090 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:13.333054 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:15:19.083362 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:19.083329 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:15:21.043649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.043613 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89"] Oct 30 18:15:21.044147 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.043862 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" podUID="28925fa7-e5d2-45c5-98ec-d0183301fd7f" containerName="kube-rbac-proxy" containerID="cri-o://9110306a3b1bee5e93e9be0b60ff41dd03cf8c8d7c98feee8b24094c5471307e" gracePeriod=30 Oct 30 18:15:21.239950 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.239916 5014 generic.go:356] "Generic (PLEG): container finished" podID="28925fa7-e5d2-45c5-98ec-d0183301fd7f" containerID="9110306a3b1bee5e93e9be0b60ff41dd03cf8c8d7c98feee8b24094c5471307e" exitCode=0 Oct 30 18:15:21.240082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.239989 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" event={"ID":"28925fa7-e5d2-45c5-98ec-d0183301fd7f","Type":"ContainerDied","Data":"9110306a3b1bee5e93e9be0b60ff41dd03cf8c8d7c98feee8b24094c5471307e"} Oct 30 18:15:21.290663 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.290643 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:15:21.399076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.399047 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-rbac-proxy-config\") pod \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " Oct 30 18:15:21.399076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.399080 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-ca\") pod \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " Oct 30 18:15:21.399339 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.399161 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-tls\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-tls\") pod \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " Oct 30 18:15:21.399339 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.399203 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-api-access-t4lkd\" (UniqueName: \"kubernetes.io/projected/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-api-access-t4lkd\") pod \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " Oct 30 18:15:21.399339 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.399260 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-client-cert\") pod \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\" (UID: \"28925fa7-e5d2-45c5-98ec-d0183301fd7f\") " Oct 30 18:15:21.399542 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.399513 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-ca" pod "28925fa7-e5d2-45c5-98ec-d0183301fd7f" (UID: "28925fa7-e5d2-45c5-98ec-d0183301fd7f"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:21.401529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.401502 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-api-access-t4lkd" pod "28925fa7-e5d2-45c5-98ec-d0183301fd7f" (UID: "28925fa7-e5d2-45c5-98ec-d0183301fd7f"). InnerVolumeSpecName "kube-api-access-t4lkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:21.401529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.401497 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-tls" pod "28925fa7-e5d2-45c5-98ec-d0183301fd7f" (UID: "28925fa7-e5d2-45c5-98ec-d0183301fd7f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:21.401741 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.401584 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-rbac-proxy-config" pod "28925fa7-e5d2-45c5-98ec-d0183301fd7f" (UID: "28925fa7-e5d2-45c5-98ec-d0183301fd7f"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:21.401938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.401912 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-client-cert" pod "28925fa7-e5d2-45c5-98ec-d0183301fd7f" (UID: "28925fa7-e5d2-45c5-98ec-d0183301fd7f"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:21.502630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.500942 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.502630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.500980 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4lkd\" (UniqueName: \"kubernetes.io/projected/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-api-access-t4lkd\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.502630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.500997 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.502630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.501011 5014 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28925fa7-e5d2-45c5-98ec-d0183301fd7f-kube-rbac-proxy-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.502630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.501026 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/28925fa7-e5d2-45c5-98ec-d0183301fd7f-prometheus-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.545157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.545126 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:15:21.545464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.545408 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="b18fc695-c6e6-489b-82c5-37da338a241e" containerName="alertmanager" containerID="cri-o://16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2" gracePeriod=120 Oct 30 18:15:21.545644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.545439 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="b18fc695-c6e6-489b-82c5-37da338a241e" containerName="config-reloader" containerID="cri-o://4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed" gracePeriod=120 Oct 30 18:15:21.557707 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.557681 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:15:21.558195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.557993 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="prometheus" containerID="cri-o://9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b" gracePeriod=600 Oct 30 18:15:21.558195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.558017 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="config-reloader" containerID="cri-o://d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc" gracePeriod=600 Oct 30 18:15:21.558195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.558016 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="thanos-sidecar" containerID="cri-o://0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd" gracePeriod=600 Oct 30 18:15:21.812566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.812543 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:21.904433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904400 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.904610 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904445 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.904610 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904498 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-tls-assets\" (UniqueName: \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-tls-assets\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.904610 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904528 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-secret-prometheus-web-tls-ca\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.904610 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904558 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-config\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-config\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.904800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904589 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.904800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904658 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.904800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904706 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.904800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904740 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-thanos-prometheus-http-client-file\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.904800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904772 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.905008 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904811 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-kube-api-access-4m7m2\" (UniqueName: \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-kube-api-access-4m7m2\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.905008 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904853 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.905008 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.904894 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324a7d04-26bd-4298-882d-1b77fd9532e4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/324a7d04-26bd-4298-882d-1b77fd9532e4-config-out\") pod \"324a7d04-26bd-4298-882d-1b77fd9532e4\" (UID: \"324a7d04-26bd-4298-882d-1b77fd9532e4\") " Oct 30 18:15:21.905654 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.905441 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-0" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:21.905787 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.905732 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-2" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:21.905861 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.905844 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-1" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:21.909134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.909101 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:21.909254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.909151 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:21.909254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.909187 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-thanos-prometheus-http-client-file" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:21.909440 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.909403 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-tls-assets" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:21.909551 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.909477 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-config" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:21.909809 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.909770 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-secret-prometheus-web-tls-ca" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:21.910101 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.910075 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-kube-api-access-4m7m2" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "kube-api-access-4m7m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:21.910197 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.910134 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324a7d04-26bd-4298-882d-1b77fd9532e4-config-out" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:21.911131 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.911110 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "pvc-c02e8472-bb1a-4974-aa89-86eef715889b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:15:21.921611 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:21.921576 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config" pod "324a7d04-26bd-4298-882d-1b77fd9532e4" (UID: "324a7d04-26bd-4298-882d-1b77fd9532e4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.006428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006374 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.006428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006397 5014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.006428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006408 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.006428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006419 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.006428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006428 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.006682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006437 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.006682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006446 5014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.006682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006456 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/324a7d04-26bd-4298-882d-1b77fd9532e4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.006682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006465 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m7m2\" (UniqueName: \"kubernetes.io/projected/324a7d04-26bd-4298-882d-1b77fd9532e4-kube-api-access-4m7m2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.006682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006474 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.006682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006484 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/324a7d04-26bd-4298-882d-1b77fd9532e4-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.006682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006508 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:15:22.006682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.006517 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/324a7d04-26bd-4298-882d-1b77fd9532e4-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.017190 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.017173 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c02e8472-bb1a-4974-aa89-86eef715889b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:15:22.107903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.107879 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" Oct 30 18:15:22.250124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.248418 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" Oct 30 18:15:22.250124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.248415 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89" event={"ID":"28925fa7-e5d2-45c5-98ec-d0183301fd7f","Type":"ContainerDied","Data":"7a8c696be3887e8d7ed780e9e792c0687105ef29f4ca91ff1f9aaea1ddd16b12"} Oct 30 18:15:22.250124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.248556 5014 scope.go:122] "RemoveContainer" containerID="9110306a3b1bee5e93e9be0b60ff41dd03cf8c8d7c98feee8b24094c5471307e" Oct 30 18:15:22.252078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.252048 5014 generic.go:356] "Generic (PLEG): container finished" podID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerID="0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd" exitCode=0 Oct 30 18:15:22.252078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.252076 5014 generic.go:356] "Generic (PLEG): container finished" podID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerID="d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc" exitCode=0 Oct 30 18:15:22.252329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.252088 5014 generic.go:356] "Generic (PLEG): container finished" podID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerID="9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b" exitCode=0 Oct 30 18:15:22.252329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.252227 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"324a7d04-26bd-4298-882d-1b77fd9532e4","Type":"ContainerDied","Data":"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd"} Oct 30 18:15:22.252329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.252293 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"324a7d04-26bd-4298-882d-1b77fd9532e4","Type":"ContainerDied","Data":"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc"} Oct 30 18:15:22.252498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.252345 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"324a7d04-26bd-4298-882d-1b77fd9532e4","Type":"ContainerDied","Data":"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b"} Oct 30 18:15:22.252498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.252395 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"324a7d04-26bd-4298-882d-1b77fd9532e4","Type":"ContainerDied","Data":"398f266445aa21ea567c4d760ef52ed10e9f5fb0a134fb22d5d3aaebdbd51ddd"} Oct 30 18:15:22.252646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.252625 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:22.258026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.257968 5014 generic.go:356] "Generic (PLEG): container finished" podID="b18fc695-c6e6-489b-82c5-37da338a241e" containerID="4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed" exitCode=0 Oct 30 18:15:22.258788 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.258380 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b18fc695-c6e6-489b-82c5-37da338a241e","Type":"ContainerDied","Data":"4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed"} Oct 30 18:15:22.269451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.269407 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:15:22.269657 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.269634 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" containerName="otc-container" containerID="cri-o://462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d" gracePeriod=30 Oct 30 18:15:22.294706 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.294663 5014 scope.go:122] "RemoveContainer" containerID="0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd" Oct 30 18:15:22.375435 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.375405 5014 scope.go:122] "RemoveContainer" containerID="d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc" Oct 30 18:15:22.414769 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.414746 5014 scope.go:122] "RemoveContainer" containerID="9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b" Oct 30 18:15:22.418731 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.418706 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:15:22.418916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.418897 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:15:22.430168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.430146 5014 scope.go:122] "RemoveContainer" containerID="353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b" Oct 30 18:15:22.446507 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.446418 5014 scope.go:122] "RemoveContainer" containerID="0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd" Oct 30 18:15:22.446827 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:22.446808 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd\": container with ID starting with 0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd not found: ID does not exist" containerID="0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd" Oct 30 18:15:22.446908 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.446839 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd"} err="failed to get container status \"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd\": rpc error: code = NotFound desc = could not find container \"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd\": container with ID starting with 0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd not found: ID does not exist" Oct 30 18:15:22.446908 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.446868 5014 scope.go:122] "RemoveContainer" containerID="d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc" Oct 30 18:15:22.447176 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:22.447156 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc\": container with ID starting with d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc not found: ID does not exist" containerID="d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc" Oct 30 18:15:22.447280 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.447180 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc"} err="failed to get container status \"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc\": rpc error: code = NotFound desc = could not find container \"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc\": container with ID starting with d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc not found: ID does not exist" Oct 30 18:15:22.447280 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.447196 5014 scope.go:122] "RemoveContainer" containerID="9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b" Oct 30 18:15:22.447482 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:22.447464 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b\": container with ID starting with 9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b not found: ID does not exist" containerID="9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b" Oct 30 18:15:22.447540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.447488 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b"} err="failed to get container status \"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b\": rpc error: code = NotFound desc = could not find container \"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b\": container with ID starting with 9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b not found: ID does not exist" Oct 30 18:15:22.447540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.447504 5014 scope.go:122] "RemoveContainer" containerID="353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b" Oct 30 18:15:22.447861 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:22.447839 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b\": container with ID starting with 353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b not found: ID does not exist" containerID="353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b" Oct 30 18:15:22.447967 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.447871 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b"} err="failed to get container status \"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b\": rpc error: code = NotFound desc = could not find container \"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b\": container with ID starting with 353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b not found: ID does not exist" Oct 30 18:15:22.447967 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.447892 5014 scope.go:122] "RemoveContainer" containerID="0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd" Oct 30 18:15:22.448184 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.448165 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd"} err="failed to get container status \"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd\": rpc error: code = NotFound desc = could not find container \"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd\": container with ID starting with 0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd not found: ID does not exist" Oct 30 18:15:22.448265 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.448185 5014 scope.go:122] "RemoveContainer" containerID="d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc" Oct 30 18:15:22.448442 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.448419 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc"} err="failed to get container status \"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc\": rpc error: code = NotFound desc = could not find container \"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc\": container with ID starting with d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc not found: ID does not exist" Oct 30 18:15:22.448498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.448444 5014 scope.go:122] "RemoveContainer" containerID="9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b" Oct 30 18:15:22.448761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.448740 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b"} err="failed to get container status \"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b\": rpc error: code = NotFound desc = could not find container \"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b\": container with ID starting with 9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b not found: ID does not exist" Oct 30 18:15:22.448850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.448762 5014 scope.go:122] "RemoveContainer" containerID="353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b" Oct 30 18:15:22.449055 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.449036 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b"} err="failed to get container status \"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b\": rpc error: code = NotFound desc = could not find container \"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b\": container with ID starting with 353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b not found: ID does not exist" Oct 30 18:15:22.449118 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.449056 5014 scope.go:122] "RemoveContainer" containerID="0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd" Oct 30 18:15:22.449389 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.449361 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd"} err="failed to get container status \"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd\": rpc error: code = NotFound desc = could not find container \"0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd\": container with ID starting with 0420da288c18dd21fa58c8064405be9afe3e1c2c2b6d7128ef9a06c92faedebd not found: ID does not exist" Oct 30 18:15:22.449389 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.449387 5014 scope.go:122] "RemoveContainer" containerID="d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc" Oct 30 18:15:22.450251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.450224 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc"} err="failed to get container status \"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc\": rpc error: code = NotFound desc = could not find container \"d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc\": container with ID starting with d8714e9282fd279b3e56dc57d7860308d29666006bc6b72fd0c661be6b2ccfdc not found: ID does not exist" Oct 30 18:15:22.450251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.450250 5014 scope.go:122] "RemoveContainer" containerID="9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b" Oct 30 18:15:22.450512 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.450491 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b"} err="failed to get container status \"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b\": rpc error: code = NotFound desc = could not find container \"9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b\": container with ID starting with 9711459b3d04561393b24804460189aa4368d530c7dbf3304994434e2807017b not found: ID does not exist" Oct 30 18:15:22.450575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.450513 5014 scope.go:122] "RemoveContainer" containerID="353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b" Oct 30 18:15:22.450785 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.450737 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b"} err="failed to get container status \"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b\": rpc error: code = NotFound desc = could not find container \"353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b\": container with ID starting with 353f4e51c7833d6f5ad9df6a85727641652c54b4e85739a1393c4dfed303158b not found: ID does not exist" Oct 30 18:15:22.451523 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.451502 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89"] Oct 30 18:15:22.457152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.457132 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-z6v89"] Oct 30 18:15:22.473297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.473272 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:22.560768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.560746 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:22.617274 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617247 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-alertmanager-data-science-monitoringstack-db\") pod \"b18fc695-c6e6-489b-82c5-37da338a241e\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " Oct 30 18:15:22.617459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617291 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-tls-certs\") pod \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " Oct 30 18:15:22.617459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617311 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-config-volume\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-config-volume\") pod \"b18fc695-c6e6-489b-82c5-37da338a241e\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " Oct 30 18:15:22.617459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617344 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-monitoring-tls-certs\") pod \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " Oct 30 18:15:22.617459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617451 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otlp-tls-certs\") pod \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " Oct 30 18:15:22.617725 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617509 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-data-science-collector-ta-client-cert\") pod \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " Oct 30 18:15:22.617725 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617551 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-kube-api-access-p8wl4\" (UniqueName: \"kubernetes.io/projected/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-kube-api-access-p8wl4\") pod \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " Oct 30 18:15:22.617725 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617576 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-cluster-tls-config\") pod \"b18fc695-c6e6-489b-82c5-37da338a241e\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " Oct 30 18:15:22.617725 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617638 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-kube-api-access-m55hq\" (UniqueName: \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-kube-api-access-m55hq\") pod \"b18fc695-c6e6-489b-82c5-37da338a241e\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " Oct 30 18:15:22.617725 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617682 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-config-out\") pod \"b18fc695-c6e6-489b-82c5-37da338a241e\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " Oct 30 18:15:22.617725 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617691 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-alertmanager-data-science-monitoringstack-db" pod "b18fc695-c6e6-489b-82c5-37da338a241e" (UID: "b18fc695-c6e6-489b-82c5-37da338a241e"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:22.618016 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617778 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-web-config\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-web-config\") pod \"b18fc695-c6e6-489b-82c5-37da338a241e\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " Oct 30 18:15:22.618016 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617829 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otc-internal\" (UniqueName: \"kubernetes.io/configmap/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otc-internal\") pod \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\" (UID: \"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60\") " Oct 30 18:15:22.618016 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.617861 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-tls-assets\" (UniqueName: \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-tls-assets\") pod \"b18fc695-c6e6-489b-82c5-37da338a241e\" (UID: \"b18fc695-c6e6-489b-82c5-37da338a241e\") " Oct 30 18:15:22.619522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.618142 5014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.619522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.619309 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otc-internal" pod "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" (UID: "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:22.620286 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.620243 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-config-volume" pod "b18fc695-c6e6-489b-82c5-37da338a241e" (UID: "b18fc695-c6e6-489b-82c5-37da338a241e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.620386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.620356 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-tls-certs" pod "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" (UID: "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.620705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.620681 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-monitoring-tls-certs" pod "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" (UID: "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.620705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.620694 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-kube-api-access-p8wl4" pod "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" (UID: "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60"). InnerVolumeSpecName "kube-api-access-p8wl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:22.621857 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.621830 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-kube-api-access-m55hq" pod "b18fc695-c6e6-489b-82c5-37da338a241e" (UID: "b18fc695-c6e6-489b-82c5-37da338a241e"). InnerVolumeSpecName "kube-api-access-m55hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:22.622467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.622424 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-tls-assets" pod "b18fc695-c6e6-489b-82c5-37da338a241e" (UID: "b18fc695-c6e6-489b-82c5-37da338a241e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:22.622860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.622837 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-data-science-collector-ta-client-cert" pod "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" (UID: "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.623031 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.623015 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otlp-tls-certs" pod "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" (UID: "02f3c362-e4dc-4711-bfe3-90e3aa5f1d60"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.623482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.623465 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-config-out" pod "b18fc695-c6e6-489b-82c5-37da338a241e" (UID: "b18fc695-c6e6-489b-82c5-37da338a241e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:22.627877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.627854 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-cluster-tls-config" pod "b18fc695-c6e6-489b-82c5-37da338a241e" (UID: "b18fc695-c6e6-489b-82c5-37da338a241e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.635517 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.635495 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-web-config" pod "b18fc695-c6e6-489b-82c5-37da338a241e" (UID: "b18fc695-c6e6-489b-82c5-37da338a241e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.719270 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719243 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.719270 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719267 5014 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-config-volume\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.719416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719278 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.719416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719287 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.719416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719295 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.719416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719306 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8wl4\" (UniqueName: \"kubernetes.io/projected/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-kube-api-access-p8wl4\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.719416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719317 5014 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-cluster-tls-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.719416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719326 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m55hq\" (UniqueName: \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-kube-api-access-m55hq\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.719416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719334 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b18fc695-c6e6-489b-82c5-37da338a241e-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.719416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719342 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b18fc695-c6e6-489b-82c5-37da338a241e-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.719416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719350 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.719416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:22.719357 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b18fc695-c6e6-489b-82c5-37da338a241e-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:23.016468 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.016381 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:15:23.016665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.016633 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo" containerID="cri-o://f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7" gracePeriod=30 Oct 30 18:15:23.016920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.016724 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo-gateway" containerID="cri-o://54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5" gracePeriod=30 Oct 30 18:15:23.016995 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.016919 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo-gateway-opa" containerID="cri-o://ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d" gracePeriod=30 Oct 30 18:15:23.141536 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.141509 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28925fa7-e5d2-45c5-98ec-d0183301fd7f" path="/var/lib/kubelet/pods/28925fa7-e5d2-45c5-98ec-d0183301fd7f/volumes" Oct 30 18:15:23.142037 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.142018 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" path="/var/lib/kubelet/pods/324a7d04-26bd-4298-882d-1b77fd9532e4/volumes" Oct 30 18:15:23.267953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.267881 5014 generic.go:356] "Generic (PLEG): container finished" podID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerID="ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d" exitCode=0 Oct 30 18:15:23.267953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.267905 5014 generic.go:356] "Generic (PLEG): container finished" podID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerID="f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7" exitCode=0 Oct 30 18:15:23.268162 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.267967 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"254b74cd-53ce-47bd-bf7d-a0d5c7b34289","Type":"ContainerDied","Data":"ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d"} Oct 30 18:15:23.268162 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.268042 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"254b74cd-53ce-47bd-bf7d-a0d5c7b34289","Type":"ContainerDied","Data":"f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7"} Oct 30 18:15:23.271955 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.271935 5014 generic.go:356] "Generic (PLEG): container finished" podID="02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" containerID="462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d" exitCode=0 Oct 30 18:15:23.272095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.272016 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:23.272160 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.272032 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60","Type":"ContainerDied","Data":"462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d"} Oct 30 18:15:23.272224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.272163 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"02f3c362-e4dc-4711-bfe3-90e3aa5f1d60","Type":"ContainerDied","Data":"6ea84ab35c40e53044569aa6474fdc1d6a5c6b50047519ab67bbdc3c3dfb3988"} Oct 30 18:15:23.272224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.272197 5014 scope.go:122] "RemoveContainer" containerID="462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d" Oct 30 18:15:23.275075 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.275054 5014 generic.go:356] "Generic (PLEG): container finished" podID="b18fc695-c6e6-489b-82c5-37da338a241e" containerID="16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2" exitCode=0 Oct 30 18:15:23.275201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.275090 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b18fc695-c6e6-489b-82c5-37da338a241e","Type":"ContainerDied","Data":"16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2"} Oct 30 18:15:23.275201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.275136 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b18fc695-c6e6-489b-82c5-37da338a241e","Type":"ContainerDied","Data":"190623d8d2862111d67af1ad8b538fec6a0e5597952c65a6a14a761e9cd8d066"} Oct 30 18:15:23.275201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.275155 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:23.282777 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.282753 5014 scope.go:122] "RemoveContainer" containerID="462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d" Oct 30 18:15:23.283200 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:23.283131 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d\": container with ID starting with 462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d not found: ID does not exist" containerID="462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d" Oct 30 18:15:23.283200 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.283165 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d"} err="failed to get container status \"462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d\": rpc error: code = NotFound desc = could not find container \"462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d\": container with ID starting with 462c5a838b770d2bf77df27ea5f2c657fb057185cd0dfa3c05bb7d108375b28d not found: ID does not exist" Oct 30 18:15:23.283200 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.283201 5014 scope.go:122] "RemoveContainer" containerID="4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed" Oct 30 18:15:23.296375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.296120 5014 scope.go:122] "RemoveContainer" containerID="16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2" Oct 30 18:15:23.309418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.309326 5014 scope.go:122] "RemoveContainer" containerID="ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175" Oct 30 18:15:23.322494 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.322474 5014 scope.go:122] "RemoveContainer" containerID="4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed" Oct 30 18:15:23.324078 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:23.324014 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed\": container with ID starting with 4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed not found: ID does not exist" containerID="4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed" Oct 30 18:15:23.324078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.324056 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed"} err="failed to get container status \"4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed\": rpc error: code = NotFound desc = could not find container \"4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed\": container with ID starting with 4c950527973c756f7d7b187456f6ffa0293e2f0f797a498fa9fb07ac528241ed not found: ID does not exist" Oct 30 18:15:23.324078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.324079 5014 scope.go:122] "RemoveContainer" containerID="16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2" Oct 30 18:15:23.324753 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:23.324727 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2\": container with ID starting with 16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2 not found: ID does not exist" containerID="16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2" Oct 30 18:15:23.324841 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.324760 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2"} err="failed to get container status \"16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2\": rpc error: code = NotFound desc = could not find container \"16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2\": container with ID starting with 16291fa3543b91aa06e7e10087234bd40d5a790e52201c74577b3abf871cc7e2 not found: ID does not exist" Oct 30 18:15:23.324841 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.324780 5014 scope.go:122] "RemoveContainer" containerID="ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175" Oct 30 18:15:23.325140 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:23.325120 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175\": container with ID starting with ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175 not found: ID does not exist" containerID="ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175" Oct 30 18:15:23.325205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.325158 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175"} err="failed to get container status \"ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175\": rpc error: code = NotFound desc = could not find container \"ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175\": container with ID starting with ed631e17ae03d6ac1446af4eeb5788f05c356393b19192dd7908c135f5164175 not found: ID does not exist" Oct 30 18:15:23.330512 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.330451 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:15:23.333785 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.333765 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:15:23.368565 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.368532 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:15:23.373680 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:23.373658 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:15:24.268502 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.268476 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:15:24.283954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.283928 5014 generic.go:356] "Generic (PLEG): container finished" podID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerID="54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5" exitCode=0 Oct 30 18:15:24.284083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.283966 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"254b74cd-53ce-47bd-bf7d-a0d5c7b34289","Type":"ContainerDied","Data":"54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5"} Oct 30 18:15:24.284083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.283988 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:15:24.284083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.284043 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"254b74cd-53ce-47bd-bf7d-a0d5c7b34289","Type":"ContainerDied","Data":"149b30bdc73f840f14f7a227ae4865e7c90f04546614c51345738d27daae2362"} Oct 30 18:15:24.284083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.284067 5014 scope.go:122] "RemoveContainer" containerID="ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d" Oct 30 18:15:24.292673 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.292656 5014 scope.go:122] "RemoveContainer" containerID="54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5" Oct 30 18:15:24.300413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.300383 5014 scope.go:122] "RemoveContainer" containerID="f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7" Oct 30 18:15:24.307550 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.307535 5014 scope.go:122] "RemoveContainer" containerID="ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d" Oct 30 18:15:24.307851 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:24.307829 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d\": container with ID starting with ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d not found: ID does not exist" containerID="ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d" Oct 30 18:15:24.307951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.307859 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d"} err="failed to get container status \"ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d\": rpc error: code = NotFound desc = could not find container \"ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d\": container with ID starting with ce5d0ed6ca953f1ae00b8784c569677f2b8060afa77880745fabcca7b8d5b88d not found: ID does not exist" Oct 30 18:15:24.307951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.307877 5014 scope.go:122] "RemoveContainer" containerID="54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5" Oct 30 18:15:24.308128 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:24.308115 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5\": container with ID starting with 54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5 not found: ID does not exist" containerID="54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5" Oct 30 18:15:24.308166 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.308132 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5"} err="failed to get container status \"54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5\": rpc error: code = NotFound desc = could not find container \"54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5\": container with ID starting with 54a807f6ed8be857a100fc32ee1db9f2341334752295566198a4dffd6307e1f5 not found: ID does not exist" Oct 30 18:15:24.308166 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.308144 5014 scope.go:122] "RemoveContainer" containerID="f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7" Oct 30 18:15:24.308352 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:24.308334 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7\": container with ID starting with f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7 not found: ID does not exist" containerID="f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7" Oct 30 18:15:24.308404 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.308361 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7"} err="failed to get container status \"f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7\": rpc error: code = NotFound desc = could not find container \"f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7\": container with ID starting with f9b4cae8a5148a1a74bda2e4d6b8156eefdebe91579b7627f1fec1dc267d45b7 not found: ID does not exist" Oct 30 18:15:24.332050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332026 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " Oct 30 18:15:24.332128 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332076 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " Oct 30 18:15:24.332128 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332099 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-tenants\") pod \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " Oct 30 18:15:24.332235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332137 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-mtls\") pod \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " Oct 30 18:15:24.332292 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332238 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " Oct 30 18:15:24.332292 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332267 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-rbac\") pod \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " Oct 30 18:15:24.332412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332312 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-kube-api-access-szgln\" (UniqueName: \"kubernetes.io/projected/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-kube-api-access-szgln\") pod \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " Oct 30 18:15:24.332412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332357 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-ca-bundle\") pod \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " Oct 30 18:15:24.332412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332392 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-conf\") pod \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\" (UID: \"254b74cd-53ce-47bd-bf7d-a0d5c7b34289\") " Oct 30 18:15:24.332567 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332408 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-serving-cabundle" pod "254b74cd-53ce-47bd-bf7d-a0d5c7b34289" (UID: "254b74cd-53ce-47bd-bf7d-a0d5c7b34289"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:24.332730 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332704 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.332835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332812 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-ca-bundle" pod "254b74cd-53ce-47bd-bf7d-a0d5c7b34289" (UID: "254b74cd-53ce-47bd-bf7d-a0d5c7b34289"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:24.333021 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.332984 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-conf" pod "254b74cd-53ce-47bd-bf7d-a0d5c7b34289" (UID: "254b74cd-53ce-47bd-bf7d-a0d5c7b34289"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:24.333111 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.333086 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-rbac" pod "254b74cd-53ce-47bd-bf7d-a0d5c7b34289" (UID: "254b74cd-53ce-47bd-bf7d-a0d5c7b34289"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:24.334441 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.334414 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "254b74cd-53ce-47bd-bf7d-a0d5c7b34289" (UID: "254b74cd-53ce-47bd-bf7d-a0d5c7b34289"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:24.334698 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.334671 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-mtls" pod "254b74cd-53ce-47bd-bf7d-a0d5c7b34289" (UID: "254b74cd-53ce-47bd-bf7d-a0d5c7b34289"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:24.334953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.334934 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-tenants" pod "254b74cd-53ce-47bd-bf7d-a0d5c7b34289" (UID: "254b74cd-53ce-47bd-bf7d-a0d5c7b34289"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:24.335062 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.335042 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-kube-api-access-szgln" pod "254b74cd-53ce-47bd-bf7d-a0d5c7b34289" (UID: "254b74cd-53ce-47bd-bf7d-a0d5c7b34289"). InnerVolumeSpecName "kube-api-access-szgln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:24.335505 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.335484 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "254b74cd-53ce-47bd-bf7d-a0d5c7b34289" (UID: "254b74cd-53ce-47bd-bf7d-a0d5c7b34289"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:15:24.433924 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.433863 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.433924 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.433888 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.433924 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.433899 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.433924 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.433924 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:15:24.434124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.433935 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-gateway-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.434124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.433943 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szgln\" (UniqueName: \"kubernetes.io/projected/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-kube-api-access-szgln\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.434124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.433952 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.434124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.433960 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/254b74cd-53ce-47bd-bf7d-a0d5c7b34289-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.442437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.442423 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:15:24.534893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.534868 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" Oct 30 18:15:24.628132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.628102 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:15:24.639712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:24.639691 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:15:25.142408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:25.142375 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" path="/var/lib/kubelet/pods/02f3c362-e4dc-4711-bfe3-90e3aa5f1d60/volumes" Oct 30 18:15:25.142778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:25.142763 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" path="/var/lib/kubelet/pods/254b74cd-53ce-47bd-bf7d-a0d5c7b34289/volumes" Oct 30 18:15:25.143181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:25.143169 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b18fc695-c6e6-489b-82c5-37da338a241e" path="/var/lib/kubelet/pods/b18fc695-c6e6-489b-82c5-37da338a241e/volumes" Oct 30 18:15:32.414214 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414178 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw"] Oct 30 18:15:32.414758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414667 5014 cpu_manager.go:414] "containerMap: removing container" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="init-config-reloader" Oct 30 18:15:32.414758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414685 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="init-config-reloader" Oct 30 18:15:32.414758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414713 5014 cpu_manager.go:414] "containerMap: removing container" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo" Oct 30 18:15:32.414758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414722 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo" Oct 30 18:15:32.414758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414733 5014 cpu_manager.go:414] "containerMap: removing container" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="prometheus" Oct 30 18:15:32.414758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414739 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="prometheus" Oct 30 18:15:32.414758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414749 5014 cpu_manager.go:414] "containerMap: removing container" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="config-reloader" Oct 30 18:15:32.414758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414755 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="config-reloader" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414768 5014 cpu_manager.go:414] "containerMap: removing container" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo-gateway" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414774 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo-gateway" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414787 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b18fc695-c6e6-489b-82c5-37da338a241e" containerName="alertmanager" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414794 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18fc695-c6e6-489b-82c5-37da338a241e" containerName="alertmanager" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414806 5014 cpu_manager.go:414] "containerMap: removing container" podUID="28925fa7-e5d2-45c5-98ec-d0183301fd7f" containerName="kube-rbac-proxy" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414813 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28925fa7-e5d2-45c5-98ec-d0183301fd7f" containerName="kube-rbac-proxy" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414826 5014 cpu_manager.go:414] "containerMap: removing container" podUID="02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" containerName="otc-container" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414832 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" containerName="otc-container" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414848 5014 cpu_manager.go:414] "containerMap: removing container" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="thanos-sidecar" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414856 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="thanos-sidecar" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414869 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b18fc695-c6e6-489b-82c5-37da338a241e" containerName="init-config-reloader" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414875 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18fc695-c6e6-489b-82c5-37da338a241e" containerName="init-config-reloader" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414894 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b18fc695-c6e6-489b-82c5-37da338a241e" containerName="config-reloader" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414901 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18fc695-c6e6-489b-82c5-37da338a241e" containerName="config-reloader" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414914 5014 cpu_manager.go:414] "containerMap: removing container" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo-gateway-opa" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.414921 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo-gateway-opa" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.415013 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.415024 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="prometheus" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.415032 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="config-reloader" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.415041 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo-gateway" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.415052 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18fc695-c6e6-489b-82c5-37da338a241e" containerName="alertmanager" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.415062 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="28925fa7-e5d2-45c5-98ec-d0183301fd7f" containerName="kube-rbac-proxy" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.415071 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f3c362-e4dc-4711-bfe3-90e3aa5f1d60" containerName="otc-container" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.415079 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="324a7d04-26bd-4298-882d-1b77fd9532e4" containerName="thanos-sidecar" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.415089 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18fc695-c6e6-489b-82c5-37da338a241e" containerName="config-reloader" Oct 30 18:15:32.415159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.415099 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="254b74cd-53ce-47bd-bf7d-a0d5c7b34289" containerName="tempo-gateway-opa" Oct 30 18:15:32.420326 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.420296 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.438669 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.438644 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:15:32.438965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.438810 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:15:32.438965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.438939 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 18:15:32.439264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.439244 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:15:32.447544 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.447527 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:15:32.447685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.447571 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-nf49j\"" Oct 30 18:15:32.448877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.448856 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw"] Oct 30 18:15:32.450541 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.450507 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 18:15:32.492986 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.492962 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.493076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.493049 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvcx2\" (UniqueName: \"kubernetes.io/projected/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-api-access-zvcx2\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.493132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.493092 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.493181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.493163 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.493216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.493202 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.594538 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.594510 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.594695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.594551 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.594695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.594586 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.594695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.594670 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.594853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.594723 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvcx2\" (UniqueName: \"kubernetes.io/projected/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-api-access-zvcx2\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.594853 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:32.594748 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 18:15:32.594853 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:32.594809 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-client-cert podName:946d98c4-7d0f-4f45-9f86-b695c9cd5b59 nodeName:}" failed. No retries permitted until 2026-10-30 18:15:33.094794674 +0000 UTC m=+1146.083135156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-x55hw" (UID: "946d98c4-7d0f-4f45-9f86-b695c9cd5b59") : secret "prometheus-operated-tls" not found Oct 30 18:15:32.595326 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.595292 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.597183 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.597163 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.597364 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.597343 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:32.643016 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:32.642986 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvcx2\" (UniqueName: \"kubernetes.io/projected/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-api-access-zvcx2\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:33.098777 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:33.098743 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:33.101435 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:33.101406 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-x55hw\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:33.333014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:33.332986 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:33.632049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:33.632014 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw"] Oct 30 18:15:33.636555 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:15:33.636491 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946d98c4_7d0f_4f45_9f86_b695c9cd5b59.slice/crio-cef51c7356d0f26417661d80a2d22cdbcc7f857f0e3d27dd1c3eb05d5235592b WatchSource:0}: Error finding container cef51c7356d0f26417661d80a2d22cdbcc7f857f0e3d27dd1c3eb05d5235592b: Status 404 returned error can't find the container with id cef51c7356d0f26417661d80a2d22cdbcc7f857f0e3d27dd1c3eb05d5235592b Oct 30 18:15:34.050202 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.050175 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:15:34.054989 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.054969 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.081853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.081827 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:15:34.081968 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.081878 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:15:34.082089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.082072 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:15:34.082192 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.082083 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:15:34.082192 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.082112 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:15:34.082192 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.082145 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:15:34.083451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.083434 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:15:34.209253 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.209215 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.209422 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.209282 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.209422 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.209392 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kn7d\" (UniqueName: \"kubernetes.io/projected/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-kube-api-access-2kn7d\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.209566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.209429 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.209566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.209457 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.209566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.209542 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.309905 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.309885 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.310029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.309917 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.310029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.309943 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.310029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.309971 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kn7d\" (UniqueName: \"kubernetes.io/projected/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-kube-api-access-2kn7d\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.310200 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.310165 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.310254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.310204 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.310254 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:34.310243 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 30 18:15:34.310368 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:34.310296 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otlp-tls-certs podName:e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1 nodeName:}" failed. No retries permitted until 2026-10-30 18:15:34.81027656 +0000 UTC m=+1147.798617044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1") : secret "data-science-collector-collector-headless-tls" not found Oct 30 18:15:34.310889 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.310860 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.312385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.312359 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.312558 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.312539 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.312701 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.312682 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.334041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.334013 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kn7d\" (UniqueName: \"kubernetes.io/projected/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-kube-api-access-2kn7d\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.335379 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.335354 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" event={"ID":"946d98c4-7d0f-4f45-9f86-b695c9cd5b59","Type":"ContainerStarted","Data":"19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3"} Oct 30 18:15:34.335512 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.335400 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" event={"ID":"946d98c4-7d0f-4f45-9f86-b695c9cd5b59","Type":"ContainerStarted","Data":"cef51c7356d0f26417661d80a2d22cdbcc7f857f0e3d27dd1c3eb05d5235592b"} Oct 30 18:15:34.335512 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.335441 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:34.376325 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.376273 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" podStartSLOduration=2.376258231 podStartE2EDuration="2.376258231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:34.375929546 +0000 UTC m=+1147.364270035" watchObservedRunningTime="2026-10-30 18:15:34.376258231 +0000 UTC m=+1147.364598718" Oct 30 18:15:34.814784 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.814746 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.817267 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.817242 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:34.969912 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:34.969872 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:15:35.267532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:35.267451 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:15:35.271927 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:15:35.271892 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e17eb0_bb45_4b8d_b8ba_c6563a232ed1.slice/crio-fcb06d250072d00f0a98ae4a061bd969655df203f23a4fd24865fc3c594078a7 WatchSource:0}: Error finding container fcb06d250072d00f0a98ae4a061bd969655df203f23a4fd24865fc3c594078a7: Status 404 returned error can't find the container with id fcb06d250072d00f0a98ae4a061bd969655df203f23a4fd24865fc3c594078a7 Oct 30 18:15:35.274169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:35.274142 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:15:35.274237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:35.274200 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:15:35.341075 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:35.341043 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1","Type":"ContainerStarted","Data":"72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81"} Oct 30 18:15:35.341187 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:35.341101 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1","Type":"ContainerStarted","Data":"fcb06d250072d00f0a98ae4a061bd969655df203f23a4fd24865fc3c594078a7"} Oct 30 18:15:35.385047 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:35.384989 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.384976597 podStartE2EDuration="1.384976597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:35.384670853 +0000 UTC m=+1148.373011341" watchObservedRunningTime="2026-10-30 18:15:35.384976597 +0000 UTC m=+1148.373317084" Oct 30 18:15:40.343389 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:40.343360 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:15:40.995714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:40.995679 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58"] Oct 30 18:15:40.999386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:40.999367 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" Oct 30 18:15:41.018314 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.018288 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58"] Oct 30 18:15:41.027827 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.027797 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-5mtd6\"" Oct 30 18:15:41.162315 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.162287 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9q26\" (UniqueName: \"kubernetes.io/projected/03e12818-2b17-4a08-a5ce-7acafced4ed2-kube-api-access-b9q26\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58\" (UID: \"03e12818-2b17-4a08-a5ce-7acafced4ed2\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" Oct 30 18:15:41.263712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.263644 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9q26\" (UniqueName: \"kubernetes.io/projected/03e12818-2b17-4a08-a5ce-7acafced4ed2-kube-api-access-b9q26\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58\" (UID: \"03e12818-2b17-4a08-a5ce-7acafced4ed2\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" Oct 30 18:15:41.286564 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.286534 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9q26\" (UniqueName: \"kubernetes.io/projected/03e12818-2b17-4a08-a5ce-7acafced4ed2-kube-api-access-b9q26\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58\" (UID: \"03e12818-2b17-4a08-a5ce-7acafced4ed2\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" Oct 30 18:15:41.327266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.327244 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" Oct 30 18:15:41.626239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.626172 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58"] Oct 30 18:15:41.629844 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:15:41.629816 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03e12818_2b17_4a08_a5ce_7acafced4ed2.slice/crio-83bcf91da07152f54e0f0343bfca6aaf3da3d8438ff6c1f7a27b73af3eb36ece WatchSource:0}: Error finding container 83bcf91da07152f54e0f0343bfca6aaf3da3d8438ff6c1f7a27b73af3eb36ece: Status 404 returned error can't find the container with id 83bcf91da07152f54e0f0343bfca6aaf3da3d8438ff6c1f7a27b73af3eb36ece Oct 30 18:15:41.819373 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.819333 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:15:41.823182 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.823158 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:41.840359 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.840335 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:15:41.840506 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.840463 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:15:41.840646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.840513 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:15:41.840646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.840527 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:15:41.840757 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.840585 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-vgnd5\"" Oct 30 18:15:41.854105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.854081 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:15:41.877708 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.877634 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:15:41.881700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.881679 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.896188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.896165 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:15:41.896385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.896363 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:15:41.896567 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.896551 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:15:41.911638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.911614 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:15:41.940473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.940455 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:15:41.940581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.940470 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:15:41.940581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.940489 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:15:41.940712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.940580 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:15:41.940712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.940647 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-ls454\"" Oct 30 18:15:41.940712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.940661 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:15:41.969067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969037 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:41.969164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969079 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.969164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969148 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.969293 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969275 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.969337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969320 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.969376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969348 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:41.969376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969371 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.969449 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969412 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.969486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969451 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrk6\" (UniqueName: \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-kube-api-access-kbrk6\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:41.969526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969515 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:41.969564 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969537 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnlnl\" (UniqueName: \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-kube-api-access-rnlnl\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.969630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969560 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.969630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969613 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.969712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969668 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:41.969712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969694 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.969787 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969721 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:41.969787 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969748 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:41.969855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969824 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:41.969888 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:41.969862 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.070196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070160 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.070196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070202 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.070337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070219 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.070337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070237 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.070337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070264 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.070337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070282 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.070337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070304 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.070337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070325 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.070337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070341 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.070743 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070475 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrk6\" (UniqueName: \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-kube-api-access-kbrk6\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.070809 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070785 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.070809 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070794 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.070912 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070821 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnlnl\" (UniqueName: \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-kube-api-access-rnlnl\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.070912 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070860 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.070912 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070890 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.071072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070918 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.071746 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.071237 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.071746 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.070935 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.071746 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.071301 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.071746 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.071331 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.071746 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.071358 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.071746 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.071371 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.071746 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.071386 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.073520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.073485 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.073828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.073644 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.073828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.073734 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.073828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.073770 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.074524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.074220 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.074524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.074373 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.074785 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.074758 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.075022 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.074999 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.075866 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.075834 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.075988 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.075972 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.076523 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.076503 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.076614 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.076581 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.076825 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.076808 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.093560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.093541 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrk6\" (UniqueName: \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-kube-api-access-kbrk6\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.097313 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.097295 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnlnl\" (UniqueName: \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-kube-api-access-rnlnl\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:42.141461 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.141413 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:42.373862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.373833 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" event={"ID":"03e12818-2b17-4a08-a5ce-7acafced4ed2","Type":"ContainerStarted","Data":"af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5"} Oct 30 18:15:42.373979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.373883 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" event={"ID":"03e12818-2b17-4a08-a5ce-7acafced4ed2","Type":"ContainerStarted","Data":"83bcf91da07152f54e0f0343bfca6aaf3da3d8438ff6c1f7a27b73af3eb36ece"} Oct 30 18:15:42.412310 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.412237 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" podStartSLOduration=2.412223577 podStartE2EDuration="2.412223577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:42.410939361 +0000 UTC m=+1155.399279850" watchObservedRunningTime="2026-10-30 18:15:42.412223577 +0000 UTC m=+1155.400564068" Oct 30 18:15:42.468790 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:42.468758 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:15:42.472175 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:15:42.472143 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06188f5d_ad56_49f7_baba_8e8b7e1154dc.slice/crio-31142d658a081f81548aa0484b1627471a8f47b15201e297d990be0760ea8cf9 WatchSource:0}: Error finding container 31142d658a081f81548aa0484b1627471a8f47b15201e297d990be0760ea8cf9: Status 404 returned error can't find the container with id 31142d658a081f81548aa0484b1627471a8f47b15201e297d990be0760ea8cf9 Oct 30 18:15:43.333510 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:15:43.333476 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:15:43.379903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:43.379867 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"06188f5d-ad56-49f7-baba-8e8b7e1154dc","Type":"ContainerStarted","Data":"68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930"} Oct 30 18:15:43.380042 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:43.379970 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"06188f5d-ad56-49f7-baba-8e8b7e1154dc","Type":"ContainerStarted","Data":"31142d658a081f81548aa0484b1627471a8f47b15201e297d990be0760ea8cf9"} Oct 30 18:15:46.394663 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:46.394571 5014 generic.go:356] "Generic (PLEG): container finished" podID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerID="68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930" exitCode=0 Oct 30 18:15:46.394663 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:46.394644 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"06188f5d-ad56-49f7-baba-8e8b7e1154dc","Type":"ContainerDied","Data":"68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930"} Oct 30 18:15:47.401178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:47.401150 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"06188f5d-ad56-49f7-baba-8e8b7e1154dc","Type":"ContainerStarted","Data":"c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509"} Oct 30 18:15:47.401574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:47.401201 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"06188f5d-ad56-49f7-baba-8e8b7e1154dc","Type":"ContainerStarted","Data":"4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed"} Oct 30 18:15:47.401574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:47.401382 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:47.471098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:47.471057 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.185900829 podStartE2EDuration="6.47104482s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.285143991s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:47.45828752 +0000 UTC m=+1160.446628025" watchObservedRunningTime="2026-10-30 18:15:47.47104482 +0000 UTC m=+1160.459385307" Oct 30 18:15:49.932438 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:49.932405 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:49.960637 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:49.960613 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:50.033308 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:50.033282 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:50.033436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:50.033417 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:50.047184 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:50.047164 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") DevicePath \"csi-3ece98a829a66f7cef6e6b4403002493a48d56f442e1673d82fff8fda237a009\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:50.148959 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:50.148924 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3d0cf4484736f6d40d16a5331fa2b97b9730899f1120a88b3f09811f40fc91e0/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:50.175742 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:50.175714 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:50.300508 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:50.300453 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:50.626969 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:50.626940 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:15:50.629295 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:15:50.629271 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cf0339_a1d5_4b2c_abeb_01dd1d27bb3c.slice/crio-61e1c42efa1471383a00ec5f5ad00a10cbcb47e1ac0bf93fc514f7d869923b5c WatchSource:0}: Error finding container 61e1c42efa1471383a00ec5f5ad00a10cbcb47e1ac0bf93fc514f7d869923b5c: Status 404 returned error can't find the container with id 61e1c42efa1471383a00ec5f5ad00a10cbcb47e1ac0bf93fc514f7d869923b5c Oct 30 18:15:51.420693 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:51.420662 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c","Type":"ContainerStarted","Data":"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d"} Oct 30 18:15:51.421045 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:51.420724 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c","Type":"ContainerStarted","Data":"61e1c42efa1471383a00ec5f5ad00a10cbcb47e1ac0bf93fc514f7d869923b5c"} Oct 30 18:15:53.407571 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:53.407545 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:54.434645 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:54.434566 5014 generic.go:356] "Generic (PLEG): container finished" podID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerID="a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d" exitCode=0 Oct 30 18:15:54.434972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:54.434640 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c","Type":"ContainerDied","Data":"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d"} Oct 30 18:15:55.441717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:55.441687 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c","Type":"ContainerStarted","Data":"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff"} Oct 30 18:15:55.442082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:55.441749 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c","Type":"ContainerStarted","Data":"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7"} Oct 30 18:15:55.442082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:55.441776 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c","Type":"ContainerStarted","Data":"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb"} Oct 30 18:15:55.517760 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:15:55.517708 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=11.433182401 podStartE2EDuration="14.517695246s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.084512845s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:55.517203212 +0000 UTC m=+1168.505543725" watchObservedRunningTime="2026-10-30 18:15:55.517695246 +0000 UTC m=+1168.506035756" Oct 30 18:16:00.300959 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:00.300917 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:16:05.301665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:05.301633 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:16:05.306715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:05.306694 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:16:05.492207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:05.492181 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:16:13.334987 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:16:13.334952 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:16:22.611988 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:22.611954 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:16:22.612618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:22.612185 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" containerName="otc-container" containerID="cri-o://72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81" gracePeriod=30 Oct 30 18:16:23.857429 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.857408 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:23.992053 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.991972 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-tls-certs\") pod \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " Oct 30 18:16:23.992053 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.992024 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-data-science-collector-ta-client-cert\") pod \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " Oct 30 18:16:23.992272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.992056 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otc-internal\") pod \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " Oct 30 18:16:23.992272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.992169 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otlp-tls-certs\") pod \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " Oct 30 18:16:23.992272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.992229 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-kube-api-access-2kn7d\" (UniqueName: \"kubernetes.io/projected/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-kube-api-access-2kn7d\") pod \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " Oct 30 18:16:23.992272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.992270 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-monitoring-tls-certs\") pod \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\" (UID: \"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1\") " Oct 30 18:16:23.992698 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.992668 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otc-internal" pod "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" (UID: "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:23.994457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.994430 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-kube-api-access-2kn7d" pod "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" (UID: "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1"). InnerVolumeSpecName "kube-api-access-2kn7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:23.994803 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.994780 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-tls-certs" pod "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" (UID: "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:23.994803 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.994792 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-data-science-collector-ta-client-cert" pod "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" (UID: "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:23.994965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.994946 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-monitoring-tls-certs" pod "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" (UID: "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:23.995025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:23.994961 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otlp-tls-certs" pod "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" (UID: "e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:24.093363 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.093341 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:16:24.093363 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.093362 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:16:24.093480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.093375 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:16:24.093480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.093384 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:16:24.093480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.093393 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:16:24.093480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.093401 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kn7d\" (UniqueName: \"kubernetes.io/projected/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1-kube-api-access-2kn7d\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:16:24.577685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.577652 5014 generic.go:356] "Generic (PLEG): container finished" podID="e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" containerID="72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81" exitCode=0 Oct 30 18:16:24.577853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.577702 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.577853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.577720 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1","Type":"ContainerDied","Data":"72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81"} Oct 30 18:16:24.577853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.577772 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1","Type":"ContainerDied","Data":"fcb06d250072d00f0a98ae4a061bd969655df203f23a4fd24865fc3c594078a7"} Oct 30 18:16:24.577853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.577804 5014 scope.go:122] "RemoveContainer" containerID="72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81" Oct 30 18:16:24.587360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.587344 5014 scope.go:122] "RemoveContainer" containerID="72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81" Oct 30 18:16:24.587713 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:16:24.587683 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81\": container with ID starting with 72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81 not found: ID does not exist" containerID="72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81" Oct 30 18:16:24.587803 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.587721 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81"} err="failed to get container status \"72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81\": rpc error: code = NotFound desc = could not find container \"72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81\": container with ID starting with 72ef97e9a843e9bc0a77c3bef86081cfecfe4aa6649fdd9a6ad9115108e77e81 not found: ID does not exist" Oct 30 18:16:24.628838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.628813 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:16:24.640375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.640356 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:16:24.671740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.671714 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:16:24.672165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.672144 5014 cpu_manager.go:414] "containerMap: removing container" podUID="e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" containerName="otc-container" Oct 30 18:16:24.672165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.672164 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" containerName="otc-container" Oct 30 18:16:24.672286 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.672245 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" containerName="otc-container" Oct 30 18:16:24.676945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.676930 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.691663 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.691645 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:16:24.691774 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.691734 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:16:24.691774 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.691734 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:16:24.691774 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.691751 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:16:24.691964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.691790 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:16:24.691964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.691890 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:16:24.696666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.696645 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:16:24.800387 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.800360 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d81aace0-fa4f-482d-a390-041aff97fed0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.800495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.800393 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.800495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.800431 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfjs\" (UniqueName: \"kubernetes.io/projected/d81aace0-fa4f-482d-a390-041aff97fed0-kube-api-access-2hfjs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.800574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.800509 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.800574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.800539 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.800574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.800571 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.901776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.901720 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfjs\" (UniqueName: \"kubernetes.io/projected/d81aace0-fa4f-482d-a390-041aff97fed0-kube-api-access-2hfjs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.901776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.901753 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.901776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.901773 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.902272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.901828 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.902272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.901880 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d81aace0-fa4f-482d-a390-041aff97fed0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.902272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.901910 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.902797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.902770 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d81aace0-fa4f-482d-a390-041aff97fed0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.904412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.904386 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.904516 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.904435 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.904577 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.904514 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.904577 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.904556 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.924207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.924186 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfjs\" (UniqueName: \"kubernetes.io/projected/d81aace0-fa4f-482d-a390-041aff97fed0-kube-api-access-2hfjs\") pod \"data-science-collector-collector-0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:24.989127 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:24.989108 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:16:25.141472 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:25.141437 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1" path="/var/lib/kubelet/pods/e4e17eb0-bb45-4b8d-b8ba-c6563a232ed1/volumes" Oct 30 18:16:25.282860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:25.282837 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:16:25.285456 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:16:25.285429 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81aace0_fa4f_482d_a390_041aff97fed0.slice/crio-9d2c7bf583335e418adf875086f3d70b815d0902fff0438f4321431e5cbe85fd WatchSource:0}: Error finding container 9d2c7bf583335e418adf875086f3d70b815d0902fff0438f4321431e5cbe85fd: Status 404 returned error can't find the container with id 9d2c7bf583335e418adf875086f3d70b815d0902fff0438f4321431e5cbe85fd Oct 30 18:16:25.287651 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:25.287622 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:16:25.287743 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:25.287679 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:16:25.584950 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:25.584913 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d81aace0-fa4f-482d-a390-041aff97fed0","Type":"ContainerStarted","Data":"ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf"} Oct 30 18:16:25.585152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:25.584982 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d81aace0-fa4f-482d-a390-041aff97fed0","Type":"ContainerStarted","Data":"9d2c7bf583335e418adf875086f3d70b815d0902fff0438f4321431e5cbe85fd"} Oct 30 18:16:25.626570 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:25.626513 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.626501588 podStartE2EDuration="1.626501588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:25.626292347 +0000 UTC m=+1198.614632834" watchObservedRunningTime="2026-10-30 18:16:25.626501588 +0000 UTC m=+1198.614842074" Oct 30 18:16:27.245616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:27.245575 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:16:27.246277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:16:27.246260 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:16:43.335024 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:16:43.334982 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:17:13.334910 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:17:13.334871 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:17:43.333516 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:17:43.333477 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:17:51.035931 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.035896 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:17:51.041161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.041136 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.067056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.067032 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:17:51.067224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.067039 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:17:51.067440 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.067422 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:17:51.067554 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.067530 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:17:51.067885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.067457 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:17:51.067885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.067462 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:17:51.067885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.067506 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:17:51.067885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.067435 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-66mfz\"" Oct 30 18:17:51.070730 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.070710 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:17:51.183286 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.183258 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.183410 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.183299 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.183410 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.183319 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.183513 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.183433 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4sgj\" (UniqueName: \"kubernetes.io/projected/68011d54-a66c-42b4-aacc-6a91c27e2a7a-kube-api-access-t4sgj\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.183513 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.183492 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.183618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.183521 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.183618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.183550 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.183747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.183633 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.284551 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.284521 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.284693 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.284561 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.284693 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.284589 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.284693 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.284636 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.284693 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.284671 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4sgj\" (UniqueName: \"kubernetes.io/projected/68011d54-a66c-42b4-aacc-6a91c27e2a7a-kube-api-access-t4sgj\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.284918 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.284701 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.284918 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.284818 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.284918 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.284861 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.285368 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.285339 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.285498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.285477 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.285581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.285489 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.285831 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.285807 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.287300 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.287261 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.287735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.287713 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.287909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.287888 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:51.307845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:51.307823 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4sgj\" (UniqueName: \"kubernetes.io/projected/68011d54-a66c-42b4-aacc-6a91c27e2a7a-kube-api-access-t4sgj\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:53.092997 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:53.092965 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:17:53.093463 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:53.093165 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d81aace0-fa4f-482d-a390-041aff97fed0" containerName="otc-container" containerID="cri-o://ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf" gracePeriod=30 Oct 30 18:17:53.405306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:53.405226 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:53.433467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:53.433438 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:53.506572 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:53.506546 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:53.506730 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:53.506668 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:53.520786 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:53.520767 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:53.621850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:53.621825 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:53.648842 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:53.648822 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:53.759087 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:53.759036 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:54.081875 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:54.081852 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:17:54.084078 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:17:54.084054 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68011d54_a66c_42b4_aacc_6a91c27e2a7a.slice/crio-797f38af8416d8974578507a0ada79c74383bd1831a7e696ba40178c64ddba23 WatchSource:0}: Error finding container 797f38af8416d8974578507a0ada79c74383bd1831a7e696ba40178c64ddba23: Status 404 returned error can't find the container with id 797f38af8416d8974578507a0ada79c74383bd1831a7e696ba40178c64ddba23 Oct 30 18:17:55.006640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.006573 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"68011d54-a66c-42b4-aacc-6a91c27e2a7a","Type":"ContainerStarted","Data":"81cce074312b3a03b5dd8406170de54af007733da1649cbdc64b9118c6d73f5d"} Oct 30 18:17:55.007066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.006672 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:55.007066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.006699 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"68011d54-a66c-42b4-aacc-6a91c27e2a7a","Type":"ContainerStarted","Data":"f654ebf7c1041f518209f1845a2fda07b5e9028dedcec754a5e2afe0f1560e8c"} Oct 30 18:17:55.007066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.006724 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"68011d54-a66c-42b4-aacc-6a91c27e2a7a","Type":"ContainerStarted","Data":"ba20c962febd6887c7ff469dd5aa8b2720e8fd06cb66c1f0d12f983fc0f64430"} Oct 30 18:17:55.007066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.006747 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"68011d54-a66c-42b4-aacc-6a91c27e2a7a","Type":"ContainerStarted","Data":"797f38af8416d8974578507a0ada79c74383bd1831a7e696ba40178c64ddba23"} Oct 30 18:17:55.009556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.009534 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:55.077258 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.077053 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=4.077038482 podStartE2EDuration="4.077038482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:55.075279227 +0000 UTC m=+1288.063619728" watchObservedRunningTime="2026-10-30 18:17:55.077038482 +0000 UTC m=+1288.065378969" Oct 30 18:17:55.443832 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.443811 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:55.527469 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.527440 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-tls-certs\") pod \"d81aace0-fa4f-482d-a390-041aff97fed0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " Oct 30 18:17:55.527626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.527528 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d81aace0-fa4f-482d-a390-041aff97fed0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d81aace0-fa4f-482d-a390-041aff97fed0-otc-internal\") pod \"d81aace0-fa4f-482d-a390-041aff97fed0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " Oct 30 18:17:55.527626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.527564 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-data-science-collector-ta-client-cert\") pod \"d81aace0-fa4f-482d-a390-041aff97fed0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " Oct 30 18:17:55.527719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.527676 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81aace0-fa4f-482d-a390-041aff97fed0-kube-api-access-2hfjs\" (UniqueName: \"kubernetes.io/projected/d81aace0-fa4f-482d-a390-041aff97fed0-kube-api-access-2hfjs\") pod \"d81aace0-fa4f-482d-a390-041aff97fed0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " Oct 30 18:17:55.527761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.527723 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-otlp-tls-certs\") pod \"d81aace0-fa4f-482d-a390-041aff97fed0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " Oct 30 18:17:55.527761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.527750 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-monitoring-tls-certs\") pod \"d81aace0-fa4f-482d-a390-041aff97fed0\" (UID: \"d81aace0-fa4f-482d-a390-041aff97fed0\") " Oct 30 18:17:55.527972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.527940 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81aace0-fa4f-482d-a390-041aff97fed0-otc-internal" pod "d81aace0-fa4f-482d-a390-041aff97fed0" (UID: "d81aace0-fa4f-482d-a390-041aff97fed0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:55.528113 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.528076 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d81aace0-fa4f-482d-a390-041aff97fed0-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:17:55.529902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.529870 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-data-science-collector-ta-client-cert" pod "d81aace0-fa4f-482d-a390-041aff97fed0" (UID: "d81aace0-fa4f-482d-a390-041aff97fed0"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:55.530002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.529961 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-otlp-tls-certs" pod "d81aace0-fa4f-482d-a390-041aff97fed0" (UID: "d81aace0-fa4f-482d-a390-041aff97fed0"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:55.530002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.529970 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-monitoring-tls-certs" pod "d81aace0-fa4f-482d-a390-041aff97fed0" (UID: "d81aace0-fa4f-482d-a390-041aff97fed0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:55.530327 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.530306 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81aace0-fa4f-482d-a390-041aff97fed0-kube-api-access-2hfjs" pod "d81aace0-fa4f-482d-a390-041aff97fed0" (UID: "d81aace0-fa4f-482d-a390-041aff97fed0"). InnerVolumeSpecName "kube-api-access-2hfjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:55.530420 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.530317 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-tls-certs" pod "d81aace0-fa4f-482d-a390-041aff97fed0" (UID: "d81aace0-fa4f-482d-a390-041aff97fed0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:55.629090 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.629040 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hfjs\" (UniqueName: \"kubernetes.io/projected/d81aace0-fa4f-482d-a390-041aff97fed0-kube-api-access-2hfjs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:17:55.629090 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.629061 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:17:55.629090 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.629071 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:17:55.629090 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.629080 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:17:55.629090 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:55.629090 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d81aace0-fa4f-482d-a390-041aff97fed0-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:17:56.014572 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.014506 5014 generic.go:356] "Generic (PLEG): container finished" podID="d81aace0-fa4f-482d-a390-041aff97fed0" containerID="ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf" exitCode=0 Oct 30 18:17:56.014572 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.014550 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.014572 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.014581 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d81aace0-fa4f-482d-a390-041aff97fed0","Type":"ContainerDied","Data":"ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf"} Oct 30 18:17:56.015103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.014646 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d81aace0-fa4f-482d-a390-041aff97fed0","Type":"ContainerDied","Data":"9d2c7bf583335e418adf875086f3d70b815d0902fff0438f4321431e5cbe85fd"} Oct 30 18:17:56.015103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.014669 5014 scope.go:122] "RemoveContainer" containerID="ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf" Oct 30 18:17:56.015103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.015097 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:56.015242 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.015134 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:56.023176 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.023155 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:17:56.025379 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.025364 5014 scope.go:122] "RemoveContainer" containerID="ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf" Oct 30 18:17:56.025634 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:17:56.025617 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf\": container with ID starting with ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf not found: ID does not exist" containerID="ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf" Oct 30 18:17:56.025689 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.025642 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf"} err="failed to get container status \"ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf\": rpc error: code = NotFound desc = could not find container \"ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf\": container with ID starting with ae7ba896029f638b75a7d5debf7ea6b9d2248145c011b75665c418288b6b5cdf not found: ID does not exist" Oct 30 18:17:56.064484 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.064456 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:17:56.076103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.076078 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:17:56.106380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.106354 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:17:56.106771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.106756 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d81aace0-fa4f-482d-a390-041aff97fed0" containerName="otc-container" Oct 30 18:17:56.106852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.106772 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d81aace0-fa4f-482d-a390-041aff97fed0" containerName="otc-container" Oct 30 18:17:56.106852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.106830 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d81aace0-fa4f-482d-a390-041aff97fed0" containerName="otc-container" Oct 30 18:17:56.111403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.111385 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.125956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.125935 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:17:56.125956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.125947 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:17:56.126089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.125949 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:17:56.126089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.125960 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:17:56.126089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.125995 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 30 18:17:56.126089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.126000 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:17:56.131333 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.131313 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:17:56.233568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.233543 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.233568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.233570 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.233759 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.233608 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.233801 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.233760 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.233801 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.233781 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.233883 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.233813 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfvs\" (UniqueName: \"kubernetes.io/projected/6312dc0a-11b4-47ce-8cf7-7cc078373fad-kube-api-access-sxfvs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.334309 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.334274 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.334309 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.334309 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.334462 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.334335 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfvs\" (UniqueName: \"kubernetes.io/projected/6312dc0a-11b4-47ce-8cf7-7cc078373fad-kube-api-access-sxfvs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.334462 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.334389 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.334462 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.334418 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.334462 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.334438 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.335129 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.335110 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.336957 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.336931 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.337045 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.336935 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.337280 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.337265 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.337360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.337343 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.357090 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.357068 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfvs\" (UniqueName: \"kubernetes.io/projected/6312dc0a-11b4-47ce-8cf7-7cc078373fad-kube-api-access-sxfvs\") pod \"data-science-collector-collector-0\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.424931 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.424911 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:17:56.718196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.718168 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:17:56.720714 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:17:56.720690 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6312dc0a_11b4_47ce_8cf7_7cc078373fad.slice/crio-dd64d5f3bae11e7a7c03f5a219f8435949dcbb76466fce095c13c65a5d0cb15a WatchSource:0}: Error finding container dd64d5f3bae11e7a7c03f5a219f8435949dcbb76466fce095c13c65a5d0cb15a: Status 404 returned error can't find the container with id dd64d5f3bae11e7a7c03f5a219f8435949dcbb76466fce095c13c65a5d0cb15a Oct 30 18:17:56.723395 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.723366 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:17:56.723484 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:56.723425 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:17:57.022489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:57.022409 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6312dc0a-11b4-47ce-8cf7-7cc078373fad","Type":"ContainerStarted","Data":"902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e"} Oct 30 18:17:57.022489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:57.022472 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6312dc0a-11b4-47ce-8cf7-7cc078373fad","Type":"ContainerStarted","Data":"dd64d5f3bae11e7a7c03f5a219f8435949dcbb76466fce095c13c65a5d0cb15a"} Oct 30 18:17:57.064636 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:57.064573 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.064558427 podStartE2EDuration="1.064558427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:57.062650784 +0000 UTC m=+1290.050991271" watchObservedRunningTime="2026-10-30 18:17:57.064558427 +0000 UTC m=+1290.052898977" Oct 30 18:17:57.146354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:17:57.146313 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d81aace0-fa4f-482d-a390-041aff97fed0" path="/var/lib/kubelet/pods/d81aace0-fa4f-482d-a390-041aff97fed0/volumes" Oct 30 18:18:07.892561 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:07.892484 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:18:07.893020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:07.892709 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo" containerID="cri-o://ba20c962febd6887c7ff469dd5aa8b2720e8fd06cb66c1f0d12f983fc0f64430" gracePeriod=30 Oct 30 18:18:07.893020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:07.892761 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo-gateway" containerID="cri-o://f654ebf7c1041f518209f1845a2fda07b5e9028dedcec754a5e2afe0f1560e8c" gracePeriod=30 Oct 30 18:18:07.893020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:07.892775 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo-gateway-opa" containerID="cri-o://81cce074312b3a03b5dd8406170de54af007733da1649cbdc64b9118c6d73f5d" gracePeriod=30 Oct 30 18:18:08.076259 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:08.076233 5014 generic.go:356] "Generic (PLEG): container finished" podID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerID="81cce074312b3a03b5dd8406170de54af007733da1649cbdc64b9118c6d73f5d" exitCode=0 Oct 30 18:18:08.076366 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:08.076300 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"68011d54-a66c-42b4-aacc-6a91c27e2a7a","Type":"ContainerDied","Data":"81cce074312b3a03b5dd8406170de54af007733da1649cbdc64b9118c6d73f5d"} Oct 30 18:18:09.085845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.085809 5014 generic.go:356] "Generic (PLEG): container finished" podID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerID="f654ebf7c1041f518209f1845a2fda07b5e9028dedcec754a5e2afe0f1560e8c" exitCode=0 Oct 30 18:18:09.085845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.085843 5014 generic.go:356] "Generic (PLEG): container finished" podID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerID="ba20c962febd6887c7ff469dd5aa8b2720e8fd06cb66c1f0d12f983fc0f64430" exitCode=0 Oct 30 18:18:09.086332 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.085893 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"68011d54-a66c-42b4-aacc-6a91c27e2a7a","Type":"ContainerDied","Data":"f654ebf7c1041f518209f1845a2fda07b5e9028dedcec754a5e2afe0f1560e8c"} Oct 30 18:18:09.086332 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.085965 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"68011d54-a66c-42b4-aacc-6a91c27e2a7a","Type":"ContainerDied","Data":"ba20c962febd6887c7ff469dd5aa8b2720e8fd06cb66c1f0d12f983fc0f64430"} Oct 30 18:18:09.154370 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.154348 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:09.239468 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.239389 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-ca-bundle\") pod \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " Oct 30 18:18:09.239626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.239504 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " Oct 30 18:18:09.239626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.239544 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-mtls\") pod \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " Oct 30 18:18:09.239761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.239638 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-rbac\") pod \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " Oct 30 18:18:09.239761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.239667 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68011d54-a66c-42b4-aacc-6a91c27e2a7a-kube-api-access-t4sgj\" (UniqueName: \"kubernetes.io/projected/68011d54-a66c-42b4-aacc-6a91c27e2a7a-kube-api-access-t4sgj\") pod \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " Oct 30 18:18:09.239761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.239706 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-conf\") pod \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " Oct 30 18:18:09.239761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.239750 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " Oct 30 18:18:09.239967 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.239751 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-ca-bundle" pod "68011d54-a66c-42b4-aacc-6a91c27e2a7a" (UID: "68011d54-a66c-42b4-aacc-6a91c27e2a7a"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:09.239967 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.239801 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-tenants\") pod \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " Oct 30 18:18:09.239967 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.239859 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\" (UID: \"68011d54-a66c-42b4-aacc-6a91c27e2a7a\") " Oct 30 18:18:09.240120 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.240069 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-rbac" pod "68011d54-a66c-42b4-aacc-6a91c27e2a7a" (UID: "68011d54-a66c-42b4-aacc-6a91c27e2a7a"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:09.240178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.240135 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-conf" pod "68011d54-a66c-42b4-aacc-6a91c27e2a7a" (UID: "68011d54-a66c-42b4-aacc-6a91c27e2a7a"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:09.240898 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.240375 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:09.240898 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.240402 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:09.240898 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.240417 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:09.240898 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.240641 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-serving-cabundle" pod "68011d54-a66c-42b4-aacc-6a91c27e2a7a" (UID: "68011d54-a66c-42b4-aacc-6a91c27e2a7a"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:09.243081 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.243054 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-mtls" pod "68011d54-a66c-42b4-aacc-6a91c27e2a7a" (UID: "68011d54-a66c-42b4-aacc-6a91c27e2a7a"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:09.243203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.243150 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-tenants" pod "68011d54-a66c-42b4-aacc-6a91c27e2a7a" (UID: "68011d54-a66c-42b4-aacc-6a91c27e2a7a"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:09.243203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.243185 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "68011d54-a66c-42b4-aacc-6a91c27e2a7a" (UID: "68011d54-a66c-42b4-aacc-6a91c27e2a7a"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:18:09.243353 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.243331 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "68011d54-a66c-42b4-aacc-6a91c27e2a7a" (UID: "68011d54-a66c-42b4-aacc-6a91c27e2a7a"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:09.244272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.244245 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68011d54-a66c-42b4-aacc-6a91c27e2a7a-kube-api-access-t4sgj" pod "68011d54-a66c-42b4-aacc-6a91c27e2a7a" (UID: "68011d54-a66c-42b4-aacc-6a91c27e2a7a"). InnerVolumeSpecName "kube-api-access-t4sgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:09.341903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.341870 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-gateway-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:09.341903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.341904 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:09.342046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.341935 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:18:09.342046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.341952 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:09.342046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.341968 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4sgj\" (UniqueName: \"kubernetes.io/projected/68011d54-a66c-42b4-aacc-6a91c27e2a7a-kube-api-access-t4sgj\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:09.342046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.341984 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/68011d54-a66c-42b4-aacc-6a91c27e2a7a-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:09.352482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.352466 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:18:09.442895 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:09.442872 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" Oct 30 18:18:10.092759 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:10.092739 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:10.093104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:10.092738 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"68011d54-a66c-42b4-aacc-6a91c27e2a7a","Type":"ContainerDied","Data":"797f38af8416d8974578507a0ada79c74383bd1831a7e696ba40178c64ddba23"} Oct 30 18:18:10.093104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:10.092869 5014 scope.go:122] "RemoveContainer" containerID="81cce074312b3a03b5dd8406170de54af007733da1649cbdc64b9118c6d73f5d" Oct 30 18:18:10.101941 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:10.101924 5014 scope.go:122] "RemoveContainer" containerID="f654ebf7c1041f518209f1845a2fda07b5e9028dedcec754a5e2afe0f1560e8c" Oct 30 18:18:10.109698 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:10.109673 5014 scope.go:122] "RemoveContainer" containerID="ba20c962febd6887c7ff469dd5aa8b2720e8fd06cb66c1f0d12f983fc0f64430" Oct 30 18:18:10.155808 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:10.155783 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:18:10.167609 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:10.167573 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:18:10.197126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:10.196816 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:10.197126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:10.197025 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="6312dc0a-11b4-47ce-8cf7-7cc078373fad" containerName="otc-container" containerID="cri-o://902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e" gracePeriod=30 Oct 30 18:18:11.141738 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.141707 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" path="/var/lib/kubelet/pods/68011d54-a66c-42b4-aacc-6a91c27e2a7a/volumes" Oct 30 18:18:11.443437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.443416 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:11.558850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.558822 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otc-internal\") pod \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " Oct 30 18:18:11.558988 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.558872 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6312dc0a-11b4-47ce-8cf7-7cc078373fad-kube-api-access-sxfvs\" (UniqueName: \"kubernetes.io/projected/6312dc0a-11b4-47ce-8cf7-7cc078373fad-kube-api-access-sxfvs\") pod \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " Oct 30 18:18:11.558988 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.558936 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-monitoring-tls-certs\") pod \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " Oct 30 18:18:11.558988 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.558966 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-data-science-collector-ta-client-cert\") pod \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " Oct 30 18:18:11.559166 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.559001 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-tls-certs\") pod \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " Oct 30 18:18:11.559166 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.559053 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otlp-tls-certs\") pod \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\" (UID: \"6312dc0a-11b4-47ce-8cf7-7cc078373fad\") " Oct 30 18:18:11.559368 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.559328 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otc-internal" pod "6312dc0a-11b4-47ce-8cf7-7cc078373fad" (UID: "6312dc0a-11b4-47ce-8cf7-7cc078373fad"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:11.561342 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.561309 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-data-science-collector-ta-client-cert" pod "6312dc0a-11b4-47ce-8cf7-7cc078373fad" (UID: "6312dc0a-11b4-47ce-8cf7-7cc078373fad"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:11.561450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.561427 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-monitoring-tls-certs" pod "6312dc0a-11b4-47ce-8cf7-7cc078373fad" (UID: "6312dc0a-11b4-47ce-8cf7-7cc078373fad"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:11.561510 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.561483 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otlp-tls-certs" pod "6312dc0a-11b4-47ce-8cf7-7cc078373fad" (UID: "6312dc0a-11b4-47ce-8cf7-7cc078373fad"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:11.561795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.561768 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6312dc0a-11b4-47ce-8cf7-7cc078373fad-kube-api-access-sxfvs" pod "6312dc0a-11b4-47ce-8cf7-7cc078373fad" (UID: "6312dc0a-11b4-47ce-8cf7-7cc078373fad"). InnerVolumeSpecName "kube-api-access-sxfvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:11.561795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.561768 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-tls-certs" pod "6312dc0a-11b4-47ce-8cf7-7cc078373fad" (UID: "6312dc0a-11b4-47ce-8cf7-7cc078373fad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:11.659722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.659698 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:11.659722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.659721 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:11.659857 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.659731 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:11.659857 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.659742 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:11.659857 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.659751 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6312dc0a-11b4-47ce-8cf7-7cc078373fad-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:11.659857 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:11.659759 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxfvs\" (UniqueName: \"kubernetes.io/projected/6312dc0a-11b4-47ce-8cf7-7cc078373fad-kube-api-access-sxfvs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:12.106076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.106038 5014 generic.go:356] "Generic (PLEG): container finished" podID="6312dc0a-11b4-47ce-8cf7-7cc078373fad" containerID="902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e" exitCode=0 Oct 30 18:18:12.106238 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.106081 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.106238 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.106105 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6312dc0a-11b4-47ce-8cf7-7cc078373fad","Type":"ContainerDied","Data":"902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e"} Oct 30 18:18:12.106238 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.106145 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6312dc0a-11b4-47ce-8cf7-7cc078373fad","Type":"ContainerDied","Data":"dd64d5f3bae11e7a7c03f5a219f8435949dcbb76466fce095c13c65a5d0cb15a"} Oct 30 18:18:12.106238 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.106169 5014 scope.go:122] "RemoveContainer" containerID="902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e" Oct 30 18:18:12.115721 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.115699 5014 scope.go:122] "RemoveContainer" containerID="902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e" Oct 30 18:18:12.115998 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:18:12.115980 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e\": container with ID starting with 902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e not found: ID does not exist" containerID="902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e" Oct 30 18:18:12.116073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.116012 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e"} err="failed to get container status \"902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e\": rpc error: code = NotFound desc = could not find container \"902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e\": container with ID starting with 902710c046bd068b721afe547df8f802233440cd2288753fcbd4da045c1b544e not found: ID does not exist" Oct 30 18:18:12.154931 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.154906 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:12.166526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.166502 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:12.198048 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198025 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:12.198387 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198375 5014 cpu_manager.go:414] "containerMap: removing container" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo" Oct 30 18:18:12.198454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198388 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo" Oct 30 18:18:12.198454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198402 5014 cpu_manager.go:414] "containerMap: removing container" podUID="6312dc0a-11b4-47ce-8cf7-7cc078373fad" containerName="otc-container" Oct 30 18:18:12.198454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198407 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6312dc0a-11b4-47ce-8cf7-7cc078373fad" containerName="otc-container" Oct 30 18:18:12.198454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198416 5014 cpu_manager.go:414] "containerMap: removing container" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo-gateway" Oct 30 18:18:12.198454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198421 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo-gateway" Oct 30 18:18:12.198454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198428 5014 cpu_manager.go:414] "containerMap: removing container" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo-gateway-opa" Oct 30 18:18:12.198454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198432 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo-gateway-opa" Oct 30 18:18:12.198710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198503 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo" Oct 30 18:18:12.198710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198510 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="6312dc0a-11b4-47ce-8cf7-7cc078373fad" containerName="otc-container" Oct 30 18:18:12.198710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198516 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo-gateway" Oct 30 18:18:12.198710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.198523 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="68011d54-a66c-42b4-aacc-6a91c27e2a7a" containerName="tempo-gateway-opa" Oct 30 18:18:12.203121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.203106 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.217851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.217826 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:18:12.217851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.217827 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:18:12.217851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.217848 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:18:12.217851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.217867 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:18:12.218061 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.217875 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:18:12.218061 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.217889 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:18:12.223439 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.223420 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:12.365246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.365177 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqwx\" (UniqueName: \"kubernetes.io/projected/d905e4a8-3af2-4350-a103-96b0433ee224-kube-api-access-xnqwx\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.365246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.365223 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.365396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.365338 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.365396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.365388 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.365474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.365415 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.365474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.365445 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d905e4a8-3af2-4350-a103-96b0433ee224-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.465985 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.465956 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.466086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.466061 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.466158 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.466098 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.466158 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.466116 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.466158 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.466146 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d905e4a8-3af2-4350-a103-96b0433ee224-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.466317 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.466204 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqwx\" (UniqueName: \"kubernetes.io/projected/d905e4a8-3af2-4350-a103-96b0433ee224-kube-api-access-xnqwx\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.466882 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.466784 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d905e4a8-3af2-4350-a103-96b0433ee224-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.468727 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.468704 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.468804 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.468765 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.468947 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.468923 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.469044 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.468958 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.489752 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.489731 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqwx\" (UniqueName: \"kubernetes.io/projected/d905e4a8-3af2-4350-a103-96b0433ee224-kube-api-access-xnqwx\") pod \"data-science-collector-collector-0\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.515982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.515960 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:12.810772 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.810625 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:12.815101 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:18:12.815075 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd905e4a8_3af2_4350_a103_96b0433ee224.slice/crio-289bc71d1e7e84a8f4d4a93da92599e9097ac7dd3bbdca3ecf1763fa68cb1042 WatchSource:0}: Error finding container 289bc71d1e7e84a8f4d4a93da92599e9097ac7dd3bbdca3ecf1763fa68cb1042: Status 404 returned error can't find the container with id 289bc71d1e7e84a8f4d4a93da92599e9097ac7dd3bbdca3ecf1763fa68cb1042 Oct 30 18:18:12.817535 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.817507 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:18:12.817636 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:12.817561 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:18:13.113906 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:13.113840 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d905e4a8-3af2-4350-a103-96b0433ee224","Type":"ContainerStarted","Data":"1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e"} Oct 30 18:18:13.113906 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:13.113884 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d905e4a8-3af2-4350-a103-96b0433ee224","Type":"ContainerStarted","Data":"289bc71d1e7e84a8f4d4a93da92599e9097ac7dd3bbdca3ecf1763fa68cb1042"} Oct 30 18:18:13.141776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:13.141747 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6312dc0a-11b4-47ce-8cf7-7cc078373fad" path="/var/lib/kubelet/pods/6312dc0a-11b4-47ce-8cf7-7cc078373fad/volumes" Oct 30 18:18:13.155981 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:13.155937 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.155924332 podStartE2EDuration="1.155924332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:13.154354789 +0000 UTC m=+1306.142695279" watchObservedRunningTime="2026-10-30 18:18:13.155924332 +0000 UTC m=+1306.144264819" Oct 30 18:18:13.332475 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:18:13.332440 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:18:25.604275 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.604238 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:18:25.608453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.608432 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.625105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.625083 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:18:25.625225 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.625113 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:18:25.625225 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.625119 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:18:25.636474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.636449 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:18:25.636604 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.636493 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:18:25.636943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.636921 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:18:25.636943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.636923 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:18:25.637071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.636947 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-gh5g6\"" Oct 30 18:18:25.639628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.639577 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:18:25.775963 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.775937 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.776073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.775979 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.776073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.776001 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.776158 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.776120 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.776158 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.776149 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.776243 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.776166 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tdk6\" (UniqueName: \"kubernetes.io/projected/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-kube-api-access-8tdk6\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.776243 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.776183 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.776332 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.776269 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.877408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.877355 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.877408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.877385 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.877408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.877402 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tdk6\" (UniqueName: \"kubernetes.io/projected/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-kube-api-access-8tdk6\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.877637 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.877419 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.877637 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.877452 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.877637 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.877519 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.877637 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.877558 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.877637 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.877588 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.878182 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.878160 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.878459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.878433 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.878557 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.878201 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.878557 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.878320 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.880120 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.880100 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.880187 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.880173 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.880323 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.880294 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:25.899885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:25.899858 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tdk6\" (UniqueName: \"kubernetes.io/projected/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-kube-api-access-8tdk6\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:26.294893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.294809 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:26.295091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.295043 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d905e4a8-3af2-4350-a103-96b0433ee224" containerName="otc-container" containerID="cri-o://1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e" gracePeriod=30 Oct 30 18:18:26.535650 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.535629 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:26.685027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.684994 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-data-science-collector-ta-client-cert\") pod \"d905e4a8-3af2-4350-a103-96b0433ee224\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " Oct 30 18:18:26.686040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.685037 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-monitoring-tls-certs\") pod \"d905e4a8-3af2-4350-a103-96b0433ee224\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " Oct 30 18:18:26.686040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.685055 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d905e4a8-3af2-4350-a103-96b0433ee224-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d905e4a8-3af2-4350-a103-96b0433ee224-otc-internal\") pod \"d905e4a8-3af2-4350-a103-96b0433ee224\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " Oct 30 18:18:26.686040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.685137 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-otlp-tls-certs\") pod \"d905e4a8-3af2-4350-a103-96b0433ee224\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " Oct 30 18:18:26.686040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.685233 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d905e4a8-3af2-4350-a103-96b0433ee224-kube-api-access-xnqwx\" (UniqueName: \"kubernetes.io/projected/d905e4a8-3af2-4350-a103-96b0433ee224-kube-api-access-xnqwx\") pod \"d905e4a8-3af2-4350-a103-96b0433ee224\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " Oct 30 18:18:26.686040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.685271 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-tls-certs\") pod \"d905e4a8-3af2-4350-a103-96b0433ee224\" (UID: \"d905e4a8-3af2-4350-a103-96b0433ee224\") " Oct 30 18:18:26.686040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.685566 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d905e4a8-3af2-4350-a103-96b0433ee224-otc-internal" pod "d905e4a8-3af2-4350-a103-96b0433ee224" (UID: "d905e4a8-3af2-4350-a103-96b0433ee224"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:26.686040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.685914 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d905e4a8-3af2-4350-a103-96b0433ee224-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:26.687619 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.687581 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-otlp-tls-certs" pod "d905e4a8-3af2-4350-a103-96b0433ee224" (UID: "d905e4a8-3af2-4350-a103-96b0433ee224"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:26.687952 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.687934 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-monitoring-tls-certs" pod "d905e4a8-3af2-4350-a103-96b0433ee224" (UID: "d905e4a8-3af2-4350-a103-96b0433ee224"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:26.688213 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.688189 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-data-science-collector-ta-client-cert" pod "d905e4a8-3af2-4350-a103-96b0433ee224" (UID: "d905e4a8-3af2-4350-a103-96b0433ee224"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:26.688340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.688276 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d905e4a8-3af2-4350-a103-96b0433ee224-kube-api-access-xnqwx" pod "d905e4a8-3af2-4350-a103-96b0433ee224" (UID: "d905e4a8-3af2-4350-a103-96b0433ee224"). InnerVolumeSpecName "kube-api-access-xnqwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:26.688340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.688276 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-tls-certs" pod "d905e4a8-3af2-4350-a103-96b0433ee224" (UID: "d905e4a8-3af2-4350-a103-96b0433ee224"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:26.786976 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.786955 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnqwx\" (UniqueName: \"kubernetes.io/projected/d905e4a8-3af2-4350-a103-96b0433ee224-kube-api-access-xnqwx\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:26.786976 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.786975 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:26.787098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.786987 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:26.787098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.786997 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:26.787098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:26.787006 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d905e4a8-3af2-4350-a103-96b0433ee224-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:27.187442 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.187419 5014 generic.go:356] "Generic (PLEG): container finished" podID="d905e4a8-3af2-4350-a103-96b0433ee224" containerID="1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e" exitCode=0 Oct 30 18:18:27.187579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.187472 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.187579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.187473 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d905e4a8-3af2-4350-a103-96b0433ee224","Type":"ContainerDied","Data":"1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e"} Oct 30 18:18:27.187712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.187619 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d905e4a8-3af2-4350-a103-96b0433ee224","Type":"ContainerDied","Data":"289bc71d1e7e84a8f4d4a93da92599e9097ac7dd3bbdca3ecf1763fa68cb1042"} Oct 30 18:18:27.187712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.187641 5014 scope.go:122] "RemoveContainer" containerID="1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e" Oct 30 18:18:27.196206 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.196188 5014 scope.go:122] "RemoveContainer" containerID="1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e" Oct 30 18:18:27.196479 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:18:27.196459 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e\": container with ID starting with 1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e not found: ID does not exist" containerID="1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e" Oct 30 18:18:27.196555 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.196487 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e"} err="failed to get container status \"1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e\": rpc error: code = NotFound desc = could not find container \"1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e\": container with ID starting with 1fdc50e598269b06d7cdb0fabb8aa9c7a70a0bed8dcd6a85dddd05ba7b75056e not found: ID does not exist" Oct 30 18:18:27.237696 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.237676 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:27.250001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.249978 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:27.281089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.281066 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:27.281627 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.281578 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d905e4a8-3af2-4350-a103-96b0433ee224" containerName="otc-container" Oct 30 18:18:27.281698 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.281630 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d905e4a8-3af2-4350-a103-96b0433ee224" containerName="otc-container" Oct 30 18:18:27.281741 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.281708 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d905e4a8-3af2-4350-a103-96b0433ee224" containerName="otc-container" Oct 30 18:18:27.286314 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.286294 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.300876 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.300859 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:18:27.300987 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.300969 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 30 18:18:27.301066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.301053 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:18:27.303772 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.303747 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:18:27.306671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.306650 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:27.390917 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.390892 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9555m\" (UniqueName: \"kubernetes.io/projected/bd4aa453-4645-409a-b1f1-e59b0123abbf-kube-api-access-9555m\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.391044 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.390931 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bd4aa453-4645-409a-b1f1-e59b0123abbf-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.391044 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.390954 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.391119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.391060 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.391119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.391081 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.391191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.391152 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.401241 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.401220 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:18:27.401374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.401358 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:18:27.492117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.492055 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.492117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.492114 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.492261 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.492159 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9555m\" (UniqueName: \"kubernetes.io/projected/bd4aa453-4645-409a-b1f1-e59b0123abbf-kube-api-access-9555m\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.492261 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.492195 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bd4aa453-4645-409a-b1f1-e59b0123abbf-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.492261 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.492222 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.492430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.492404 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.492947 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.492923 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bd4aa453-4645-409a-b1f1-e59b0123abbf-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.494804 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.494784 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.494890 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.494815 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.495167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.495148 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.495218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.495209 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.514428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.514405 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9555m\" (UniqueName: \"kubernetes.io/projected/bd4aa453-4645-409a-b1f1-e59b0123abbf-kube-api-access-9555m\") pod \"data-science-collector-collector-0\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.599694 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.599672 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:27.895947 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.895915 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:27.899035 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:18:27.899010 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4aa453_4645_409a_b1f1_e59b0123abbf.slice/crio-181bea606cc8fca7cf92d23bf498bb3cd2ff5277bfc2552eda243e2116855c93 WatchSource:0}: Error finding container 181bea606cc8fca7cf92d23bf498bb3cd2ff5277bfc2552eda243e2116855c93: Status 404 returned error can't find the container with id 181bea606cc8fca7cf92d23bf498bb3cd2ff5277bfc2552eda243e2116855c93 Oct 30 18:18:27.901247 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.901206 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:18:27.901368 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:27.901278 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:18:28.194955 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:28.194886 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"bd4aa453-4645-409a-b1f1-e59b0123abbf","Type":"ContainerStarted","Data":"d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb"} Oct 30 18:18:28.194955 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:28.194928 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"bd4aa453-4645-409a-b1f1-e59b0123abbf","Type":"ContainerStarted","Data":"181bea606cc8fca7cf92d23bf498bb3cd2ff5277bfc2552eda243e2116855c93"} Oct 30 18:18:28.237972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:28.237926 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.237914074 podStartE2EDuration="1.237914074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:28.236273515 +0000 UTC m=+1321.224614002" watchObservedRunningTime="2026-10-30 18:18:28.237914074 +0000 UTC m=+1321.226254560" Oct 30 18:18:29.142089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:29.142057 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d905e4a8-3af2-4350-a103-96b0433ee224" path="/var/lib/kubelet/pods/d905e4a8-3af2-4350-a103-96b0433ee224/volumes" Oct 30 18:18:33.543121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:33.543083 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:33.571189 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:33.571155 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:33.643894 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:33.643866 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:33.644010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:33.643991 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:33.658054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:33.658030 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:33.761131 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:33.761102 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:33.788270 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:33.788245 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:34.039848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:34.039824 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-gh5g6\"" Oct 30 18:18:34.046929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:34.046908 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:34.374830 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:18:34.374802 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f5a11c_9c0e_4c46_bfb0_92f1572ffb1c.slice/crio-77e407ff3bb1ee25dc570391c818dcd672b4835b0aeef699bfada58e39fb6056 WatchSource:0}: Error finding container 77e407ff3bb1ee25dc570391c818dcd672b4835b0aeef699bfada58e39fb6056: Status 404 returned error can't find the container with id 77e407ff3bb1ee25dc570391c818dcd672b4835b0aeef699bfada58e39fb6056 Oct 30 18:18:34.384049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:34.384022 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:18:35.228250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:35.228221 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c","Type":"ContainerStarted","Data":"6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed"} Oct 30 18:18:35.228773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:35.228268 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c","Type":"ContainerStarted","Data":"72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7"} Oct 30 18:18:35.228773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:35.228290 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c","Type":"ContainerStarted","Data":"d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6"} Oct 30 18:18:35.228773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:35.228308 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c","Type":"ContainerStarted","Data":"77e407ff3bb1ee25dc570391c818dcd672b4835b0aeef699bfada58e39fb6056"} Oct 30 18:18:35.228773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:35.228446 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:35.231032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:35.231013 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:35.297714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:35.297666 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=10.297653748 podStartE2EDuration="10.297653748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:35.285696553 +0000 UTC m=+1328.274037040" watchObservedRunningTime="2026-10-30 18:18:35.297653748 +0000 UTC m=+1328.285994235" Oct 30 18:18:36.232560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:36.232527 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:36.232965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:36.232585 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:36.240132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:36.240109 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:42.720017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:42.719982 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:18:42.720541 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:42.720294 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo" containerID="cri-o://d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6" gracePeriod=30 Oct 30 18:18:42.720541 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:42.720324 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo-gateway" containerID="cri-o://72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7" gracePeriod=30 Oct 30 18:18:42.721167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:42.720735 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo-gateway-opa" containerID="cri-o://6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed" gracePeriod=30 Oct 30 18:18:43.275930 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:43.275904 5014 generic.go:356] "Generic (PLEG): container finished" podID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerID="6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed" exitCode=0 Oct 30 18:18:43.275930 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:43.275926 5014 generic.go:356] "Generic (PLEG): container finished" podID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerID="d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6" exitCode=0 Oct 30 18:18:43.276095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:43.275983 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c","Type":"ContainerDied","Data":"6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed"} Oct 30 18:18:43.276095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:43.276031 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c","Type":"ContainerDied","Data":"d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6"} Oct 30 18:18:43.332350 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:18:43.332323 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:18:43.401409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:43.401381 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:43.401657 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:43.401632 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="bd4aa453-4645-409a-b1f1-e59b0123abbf" containerName="otc-container" containerID="cri-o://d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb" gracePeriod=30 Oct 30 18:18:43.967434 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:43.967409 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:44.125321 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.125290 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-kube-api-access-8tdk6\" (UniqueName: \"kubernetes.io/projected/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-kube-api-access-8tdk6\") pod \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " Oct 30 18:18:44.125474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.125336 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-mtls\") pod \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " Oct 30 18:18:44.125474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.125365 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " Oct 30 18:18:44.125474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.125395 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-rbac\") pod \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " Oct 30 18:18:44.125474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.125413 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-tenants\") pod \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " Oct 30 18:18:44.125474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.125458 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-conf\") pod \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " Oct 30 18:18:44.125785 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.125499 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " Oct 30 18:18:44.125785 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.125521 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " Oct 30 18:18:44.125785 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.125632 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\" (UID: \"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c\") " Oct 30 18:18:44.125958 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.125859 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-rbac" pod "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" (UID: "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:44.126021 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.125954 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:44.126239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.126084 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-ca-bundle" pod "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" (UID: "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:44.126455 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.126240 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-conf" pod "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" (UID: "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:44.126455 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.126425 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-serving-cabundle" pod "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" (UID: "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:44.127893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.127866 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" (UID: "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:44.128002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.127899 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-mtls" pod "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" (UID: "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:44.128207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.128190 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-tenants" pod "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" (UID: "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:44.128266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.128208 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-kube-api-access-8tdk6" pod "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" (UID: "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c"). InnerVolumeSpecName "kube-api-access-8tdk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:44.129012 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.128995 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" (UID: "19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:18:44.226393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.226372 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tdk6\" (UniqueName: \"kubernetes.io/projected/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-kube-api-access-8tdk6\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:44.226498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.226397 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:44.226498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.226407 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:44.226498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.226418 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-gateway-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:44.226498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.226426 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:44.226498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.226435 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:44.226498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.226444 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:44.226498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.226469 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:18:44.235947 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.235934 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:18:44.282136 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.282110 5014 generic.go:356] "Generic (PLEG): container finished" podID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerID="72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7" exitCode=0 Oct 30 18:18:44.282216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.282161 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c","Type":"ContainerDied","Data":"72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7"} Oct 30 18:18:44.282216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.282173 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:18:44.282286 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.282214 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c","Type":"ContainerDied","Data":"77e407ff3bb1ee25dc570391c818dcd672b4835b0aeef699bfada58e39fb6056"} Oct 30 18:18:44.282286 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.282246 5014 scope.go:122] "RemoveContainer" containerID="6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed" Oct 30 18:18:44.291544 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.291525 5014 scope.go:122] "RemoveContainer" containerID="72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7" Oct 30 18:18:44.299310 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.299288 5014 scope.go:122] "RemoveContainer" containerID="d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6" Oct 30 18:18:44.307066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.307045 5014 scope.go:122] "RemoveContainer" containerID="6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed" Oct 30 18:18:44.307296 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:18:44.307277 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed\": container with ID starting with 6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed not found: ID does not exist" containerID="6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed" Oct 30 18:18:44.307366 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.307303 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed"} err="failed to get container status \"6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed\": rpc error: code = NotFound desc = could not find container \"6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed\": container with ID starting with 6d8b60a0a54c9d75a597693af6c1c9398269aa5df3d21576cc30a74bef27ffed not found: ID does not exist" Oct 30 18:18:44.307366 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.307319 5014 scope.go:122] "RemoveContainer" containerID="72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7" Oct 30 18:18:44.307542 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:18:44.307525 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7\": container with ID starting with 72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7 not found: ID does not exist" containerID="72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7" Oct 30 18:18:44.307586 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.307546 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7"} err="failed to get container status \"72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7\": rpc error: code = NotFound desc = could not find container \"72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7\": container with ID starting with 72c8df9b7e87b41012d9f049eba88ff440b3a888017c96897950e8cf24239ed7 not found: ID does not exist" Oct 30 18:18:44.307586 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.307564 5014 scope.go:122] "RemoveContainer" containerID="d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6" Oct 30 18:18:44.307816 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:18:44.307797 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6\": container with ID starting with d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6 not found: ID does not exist" containerID="d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6" Oct 30 18:18:44.307870 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.307826 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6"} err="failed to get container status \"d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6\": rpc error: code = NotFound desc = could not find container \"d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6\": container with ID starting with d867f1a91a075d5014c49e9e2b282d2d5353054548400cb66480941e342048b6 not found: ID does not exist" Oct 30 18:18:44.327404 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.327386 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" Oct 30 18:18:44.334142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.334123 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:18:44.345521 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:44.345503 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:18:45.140968 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.140904 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" path="/var/lib/kubelet/pods/19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c/volumes" Oct 30 18:18:45.758252 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.758224 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:45.942386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.942360 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-tls-certs\") pod \"bd4aa453-4645-409a-b1f1-e59b0123abbf\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " Oct 30 18:18:45.942528 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.942401 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-data-science-collector-ta-client-cert\") pod \"bd4aa453-4645-409a-b1f1-e59b0123abbf\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " Oct 30 18:18:45.942528 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.942424 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd4aa453-4645-409a-b1f1-e59b0123abbf-otc-internal\" (UniqueName: \"kubernetes.io/configmap/bd4aa453-4645-409a-b1f1-e59b0123abbf-otc-internal\") pod \"bd4aa453-4645-409a-b1f1-e59b0123abbf\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " Oct 30 18:18:45.942528 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.942450 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-otlp-tls-certs\") pod \"bd4aa453-4645-409a-b1f1-e59b0123abbf\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " Oct 30 18:18:45.942528 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.942501 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4aa453-4645-409a-b1f1-e59b0123abbf-kube-api-access-9555m\" (UniqueName: \"kubernetes.io/projected/bd4aa453-4645-409a-b1f1-e59b0123abbf-kube-api-access-9555m\") pod \"bd4aa453-4645-409a-b1f1-e59b0123abbf\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " Oct 30 18:18:45.942766 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.942533 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-monitoring-tls-certs\") pod \"bd4aa453-4645-409a-b1f1-e59b0123abbf\" (UID: \"bd4aa453-4645-409a-b1f1-e59b0123abbf\") " Oct 30 18:18:45.942871 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.942839 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4aa453-4645-409a-b1f1-e59b0123abbf-otc-internal" pod "bd4aa453-4645-409a-b1f1-e59b0123abbf" (UID: "bd4aa453-4645-409a-b1f1-e59b0123abbf"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:45.944868 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.944838 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-monitoring-tls-certs" pod "bd4aa453-4645-409a-b1f1-e59b0123abbf" (UID: "bd4aa453-4645-409a-b1f1-e59b0123abbf"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:45.944957 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.944877 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-tls-certs" pod "bd4aa453-4645-409a-b1f1-e59b0123abbf" (UID: "bd4aa453-4645-409a-b1f1-e59b0123abbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:45.945012 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.944980 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-otlp-tls-certs" pod "bd4aa453-4645-409a-b1f1-e59b0123abbf" (UID: "bd4aa453-4645-409a-b1f1-e59b0123abbf"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:45.945364 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.945345 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-data-science-collector-ta-client-cert" pod "bd4aa453-4645-409a-b1f1-e59b0123abbf" (UID: "bd4aa453-4645-409a-b1f1-e59b0123abbf"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:45.945423 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:45.945379 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4aa453-4645-409a-b1f1-e59b0123abbf-kube-api-access-9555m" pod "bd4aa453-4645-409a-b1f1-e59b0123abbf" (UID: "bd4aa453-4645-409a-b1f1-e59b0123abbf"). InnerVolumeSpecName "kube-api-access-9555m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:46.043945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.043924 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9555m\" (UniqueName: \"kubernetes.io/projected/bd4aa453-4645-409a-b1f1-e59b0123abbf-kube-api-access-9555m\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:46.043945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.043943 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:46.044066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.043953 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:46.044066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.043965 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:46.044066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.043973 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bd4aa453-4645-409a-b1f1-e59b0123abbf-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:46.044066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.043982 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4aa453-4645-409a-b1f1-e59b0123abbf-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:18:46.293814 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.293757 5014 generic.go:356] "Generic (PLEG): container finished" podID="bd4aa453-4645-409a-b1f1-e59b0123abbf" containerID="d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb" exitCode=0 Oct 30 18:18:46.293814 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.293801 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.294164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.293842 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"bd4aa453-4645-409a-b1f1-e59b0123abbf","Type":"ContainerDied","Data":"d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb"} Oct 30 18:18:46.294164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.293887 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"bd4aa453-4645-409a-b1f1-e59b0123abbf","Type":"ContainerDied","Data":"181bea606cc8fca7cf92d23bf498bb3cd2ff5277bfc2552eda243e2116855c93"} Oct 30 18:18:46.294164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.293908 5014 scope.go:122] "RemoveContainer" containerID="d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb" Oct 30 18:18:46.303029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.303008 5014 scope.go:122] "RemoveContainer" containerID="d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb" Oct 30 18:18:46.303283 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:18:46.303266 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb\": container with ID starting with d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb not found: ID does not exist" containerID="d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb" Oct 30 18:18:46.303329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.303291 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb"} err="failed to get container status \"d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb\": rpc error: code = NotFound desc = could not find container \"d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb\": container with ID starting with d25c11b0ad07a7fa3c7fd04f72b2c56fb25f7d2ed08620b6d1e514d2e5fe30eb not found: ID does not exist" Oct 30 18:18:46.344447 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.344424 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:46.356137 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.356117 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:46.387675 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.387649 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:46.388072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388056 5014 cpu_manager.go:414] "containerMap: removing container" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo-gateway" Oct 30 18:18:46.388144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388074 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo-gateway" Oct 30 18:18:46.388144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388094 5014 cpu_manager.go:414] "containerMap: removing container" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo" Oct 30 18:18:46.388144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388102 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo" Oct 30 18:18:46.388144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388122 5014 cpu_manager.go:414] "containerMap: removing container" podUID="bd4aa453-4645-409a-b1f1-e59b0123abbf" containerName="otc-container" Oct 30 18:18:46.388144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388130 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4aa453-4645-409a-b1f1-e59b0123abbf" containerName="otc-container" Oct 30 18:18:46.388409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388146 5014 cpu_manager.go:414] "containerMap: removing container" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo-gateway-opa" Oct 30 18:18:46.388409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388155 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo-gateway-opa" Oct 30 18:18:46.388409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388269 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4aa453-4645-409a-b1f1-e59b0123abbf" containerName="otc-container" Oct 30 18:18:46.388409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388283 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo-gateway-opa" Oct 30 18:18:46.388409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388301 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo-gateway" Oct 30 18:18:46.388409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.388310 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f5a11c-9c0e-4c46-bfb0-92f1572ffb1c" containerName="tempo" Oct 30 18:18:46.392903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.392887 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.407417 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.407395 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:18:46.407500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.407427 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:18:46.407500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.407448 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:18:46.407500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.407460 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:18:46.407500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.407475 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:18:46.407500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.407481 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:18:46.419926 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.419907 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:46.548216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.548149 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.548216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.548200 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdmx\" (UniqueName: \"kubernetes.io/projected/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-kube-api-access-pqdmx\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.548364 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.548220 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.548364 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.548342 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.548428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.548389 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.548428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.548405 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.649343 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.649312 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.649446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.649369 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.649446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.649393 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.649446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.649421 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.649651 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.649490 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdmx\" (UniqueName: \"kubernetes.io/projected/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-kube-api-access-pqdmx\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.649651 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.649517 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.650141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.650066 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.652108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.652087 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.652279 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.652258 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.652279 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.652272 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.652370 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.652280 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.672819 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.672793 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdmx\" (UniqueName: \"kubernetes.io/projected/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-kube-api-access-pqdmx\") pod \"data-science-collector-collector-0\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:46.705412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:46.705393 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:18:47.002947 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:47.002917 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:18:47.005084 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:18:47.005054 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cb3e8b_45b0_4e7c_bd8e_ca2e37ef8c12.slice/crio-5e6ec47c22a220cf8cbf49a710633094fd1619c36ae5bd6d657cfb7795e6b70c WatchSource:0}: Error finding container 5e6ec47c22a220cf8cbf49a710633094fd1619c36ae5bd6d657cfb7795e6b70c: Status 404 returned error can't find the container with id 5e6ec47c22a220cf8cbf49a710633094fd1619c36ae5bd6d657cfb7795e6b70c Oct 30 18:18:47.007760 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:47.007727 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:18:47.007850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:47.007788 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:18:47.142506 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:47.142472 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd4aa453-4645-409a-b1f1-e59b0123abbf" path="/var/lib/kubelet/pods/bd4aa453-4645-409a-b1f1-e59b0123abbf/volumes" Oct 30 18:18:47.301480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:47.301413 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12","Type":"ContainerStarted","Data":"465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c"} Oct 30 18:18:47.301480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:47.301465 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12","Type":"ContainerStarted","Data":"5e6ec47c22a220cf8cbf49a710633094fd1619c36ae5bd6d657cfb7795e6b70c"} Oct 30 18:18:47.342738 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:18:47.342690 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.34267664 podStartE2EDuration="1.34267664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:47.342645077 +0000 UTC m=+1340.330985569" watchObservedRunningTime="2026-10-30 18:18:47.34267664 +0000 UTC m=+1340.331017126" Oct 30 18:19:02.515370 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:02.515328 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:02.516020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:02.515992 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" containerName="otc-container" containerID="cri-o://465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c" gracePeriod=30 Oct 30 18:19:04.866107 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:04.866086 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:04.998442 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:04.998376 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-tls-certs\") pod \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " Oct 30 18:19:04.998442 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:04.998426 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-monitoring-tls-certs\") pod \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " Oct 30 18:19:04.998642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:04.998458 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-kube-api-access-pqdmx\" (UniqueName: \"kubernetes.io/projected/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-kube-api-access-pqdmx\") pod \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " Oct 30 18:19:04.998642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:04.998531 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otlp-tls-certs\") pod \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " Oct 30 18:19:04.998642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:04.998563 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-data-science-collector-ta-client-cert\") pod \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " Oct 30 18:19:04.998642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:04.998622 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otc-internal\" (UniqueName: \"kubernetes.io/configmap/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otc-internal\") pod \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\" (UID: \"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12\") " Oct 30 18:19:04.999382 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:04.999350 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otc-internal" pod "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" (UID: "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:05.000861 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.000818 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-monitoring-tls-certs" pod "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" (UID: "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:05.000949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.000874 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-tls-certs" pod "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" (UID: "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:05.000949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.000933 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-kube-api-access-pqdmx" pod "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" (UID: "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12"). InnerVolumeSpecName "kube-api-access-pqdmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:05.001194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.001170 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-data-science-collector-ta-client-cert" pod "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" (UID: "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:05.001194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.001187 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otlp-tls-certs" pod "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" (UID: "48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:05.100149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.100112 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:05.100149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.100150 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:05.100287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.100161 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqdmx\" (UniqueName: \"kubernetes.io/projected/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-kube-api-access-pqdmx\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:05.100287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.100170 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:05.100287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.100180 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:05.100287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.100188 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:05.384574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.384542 5014 generic.go:356] "Generic (PLEG): container finished" podID="48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" containerID="465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c" exitCode=0 Oct 30 18:19:05.384715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.384619 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.384715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.384627 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12","Type":"ContainerDied","Data":"465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c"} Oct 30 18:19:05.384715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.384679 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12","Type":"ContainerDied","Data":"5e6ec47c22a220cf8cbf49a710633094fd1619c36ae5bd6d657cfb7795e6b70c"} Oct 30 18:19:05.384715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.384701 5014 scope.go:122] "RemoveContainer" containerID="465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c" Oct 30 18:19:05.393394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.393376 5014 scope.go:122] "RemoveContainer" containerID="465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c" Oct 30 18:19:05.393634 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:19:05.393616 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c\": container with ID starting with 465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c not found: ID does not exist" containerID="465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c" Oct 30 18:19:05.393713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.393642 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c"} err="failed to get container status \"465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c\": rpc error: code = NotFound desc = could not find container \"465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c\": container with ID starting with 465ad3e4bf44886c1c7dc5ffa0964a5be0c1f6a34891cd89681de5297c57575c not found: ID does not exist" Oct 30 18:19:05.434044 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.434020 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:05.445717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.445694 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:05.475653 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.475632 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:05.476020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.476005 5014 cpu_manager.go:414] "containerMap: removing container" podUID="48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" containerName="otc-container" Oct 30 18:19:05.476088 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.476023 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" containerName="otc-container" Oct 30 18:19:05.476177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.476165 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" containerName="otc-container" Oct 30 18:19:05.480535 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.480519 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.495129 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.495107 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:19:05.495225 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.495165 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:19:05.495225 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.495214 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:19:05.495297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.495232 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:19:05.495297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.495236 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-770491e3\"" Oct 30 18:19:05.495297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.495264 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:19:05.501183 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.501160 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:05.603392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.603355 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.603499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.603412 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2mms\" (UniqueName: \"kubernetes.io/projected/1513b218-4536-47d9-a12b-798ac631d4d0-kube-api-access-j2mms\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.603541 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.603505 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.603579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.603538 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.603579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.603566 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.603680 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.603665 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1513b218-4536-47d9-a12b-798ac631d4d0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.704133 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.704062 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.704133 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.704125 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.704305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.704171 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.704305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.704197 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1513b218-4536-47d9-a12b-798ac631d4d0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.704305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.704237 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.704305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.704287 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2mms\" (UniqueName: \"kubernetes.io/projected/1513b218-4536-47d9-a12b-798ac631d4d0-kube-api-access-j2mms\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.705206 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.705178 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1513b218-4536-47d9-a12b-798ac631d4d0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.706945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.706924 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.707026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.707002 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.707067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.707025 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.707067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.707052 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.726865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.726840 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2mms\" (UniqueName: \"kubernetes.io/projected/1513b218-4536-47d9-a12b-798ac631d4d0-kube-api-access-j2mms\") pod \"data-science-collector-collector-0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:05.793470 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:05.793448 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:06.088384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:06.088269 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:06.090838 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:19:06.090816 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1513b218_4536_47d9_a12b_798ac631d4d0.slice/crio-8d0c3e4a7d8e4b349dfbd10f6643ccdf08bdc26e4670b71d0fa326e488ab6e0e WatchSource:0}: Error finding container 8d0c3e4a7d8e4b349dfbd10f6643ccdf08bdc26e4670b71d0fa326e488ab6e0e: Status 404 returned error can't find the container with id 8d0c3e4a7d8e4b349dfbd10f6643ccdf08bdc26e4670b71d0fa326e488ab6e0e Oct 30 18:19:06.092941 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:06.092915 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:19:06.093014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:06.092967 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:19:06.391879 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:06.391797 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1513b218-4536-47d9-a12b-798ac631d4d0","Type":"ContainerStarted","Data":"09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb"} Oct 30 18:19:06.392034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:06.391893 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1513b218-4536-47d9-a12b-798ac631d4d0","Type":"ContainerStarted","Data":"8d0c3e4a7d8e4b349dfbd10f6643ccdf08bdc26e4670b71d0fa326e488ab6e0e"} Oct 30 18:19:06.435225 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:06.435176 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.435161122 podStartE2EDuration="1.435161122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:06.435049236 +0000 UTC m=+1359.423389747" watchObservedRunningTime="2026-10-30 18:19:06.435161122 +0000 UTC m=+1359.423501610" Oct 30 18:19:07.141662 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:07.141633 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12" path="/var/lib/kubelet/pods/48cb3e8b-45b0-4e7c-bd8e-ca2e37ef8c12/volumes" Oct 30 18:19:13.335287 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:19:13.335235 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:19:18.610002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:18.609968 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:18.610491 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:18.610210 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="1513b218-4536-47d9-a12b-798ac631d4d0" containerName="otc-container" containerID="cri-o://09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb" gracePeriod=30 Oct 30 18:19:20.954634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:20.954613 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.034354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.034289 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-data-science-collector-ta-client-cert\") pod \"1513b218-4536-47d9-a12b-798ac631d4d0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " Oct 30 18:19:21.034354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.034335 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-tls-certs\") pod \"1513b218-4536-47d9-a12b-798ac631d4d0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " Oct 30 18:19:21.034563 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.034369 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-otlp-tls-certs\") pod \"1513b218-4536-47d9-a12b-798ac631d4d0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " Oct 30 18:19:21.034563 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.034396 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1513b218-4536-47d9-a12b-798ac631d4d0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1513b218-4536-47d9-a12b-798ac631d4d0-otc-internal\") pod \"1513b218-4536-47d9-a12b-798ac631d4d0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " Oct 30 18:19:21.034563 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.034417 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-monitoring-tls-certs\") pod \"1513b218-4536-47d9-a12b-798ac631d4d0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " Oct 30 18:19:21.034563 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.034449 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1513b218-4536-47d9-a12b-798ac631d4d0-kube-api-access-j2mms\" (UniqueName: \"kubernetes.io/projected/1513b218-4536-47d9-a12b-798ac631d4d0-kube-api-access-j2mms\") pod \"1513b218-4536-47d9-a12b-798ac631d4d0\" (UID: \"1513b218-4536-47d9-a12b-798ac631d4d0\") " Oct 30 18:19:21.034897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.034868 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1513b218-4536-47d9-a12b-798ac631d4d0-otc-internal" pod "1513b218-4536-47d9-a12b-798ac631d4d0" (UID: "1513b218-4536-47d9-a12b-798ac631d4d0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:21.036807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.036777 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-otlp-tls-certs" pod "1513b218-4536-47d9-a12b-798ac631d4d0" (UID: "1513b218-4536-47d9-a12b-798ac631d4d0"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:21.036807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.036791 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1513b218-4536-47d9-a12b-798ac631d4d0-kube-api-access-j2mms" pod "1513b218-4536-47d9-a12b-798ac631d4d0" (UID: "1513b218-4536-47d9-a12b-798ac631d4d0"). InnerVolumeSpecName "kube-api-access-j2mms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:21.036951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.036911 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-monitoring-tls-certs" pod "1513b218-4536-47d9-a12b-798ac631d4d0" (UID: "1513b218-4536-47d9-a12b-798ac631d4d0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:21.037245 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.037218 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-tls-certs" pod "1513b218-4536-47d9-a12b-798ac631d4d0" (UID: "1513b218-4536-47d9-a12b-798ac631d4d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:21.037321 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.037240 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-data-science-collector-ta-client-cert" pod "1513b218-4536-47d9-a12b-798ac631d4d0" (UID: "1513b218-4536-47d9-a12b-798ac631d4d0"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:21.135579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.135558 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:21.135579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.135578 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:21.135724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.135587 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1513b218-4536-47d9-a12b-798ac631d4d0-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:21.135724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.135612 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:21.135724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.135621 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2mms\" (UniqueName: \"kubernetes.io/projected/1513b218-4536-47d9-a12b-798ac631d4d0-kube-api-access-j2mms\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:21.135724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.135630 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1513b218-4536-47d9-a12b-798ac631d4d0-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:21.471083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.471054 5014 generic.go:356] "Generic (PLEG): container finished" podID="1513b218-4536-47d9-a12b-798ac631d4d0" containerID="09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb" exitCode=0 Oct 30 18:19:21.471218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.471094 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1513b218-4536-47d9-a12b-798ac631d4d0","Type":"ContainerDied","Data":"09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb"} Oct 30 18:19:21.471218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.471107 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.471218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.471128 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1513b218-4536-47d9-a12b-798ac631d4d0","Type":"ContainerDied","Data":"8d0c3e4a7d8e4b349dfbd10f6643ccdf08bdc26e4670b71d0fa326e488ab6e0e"} Oct 30 18:19:21.471218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.471150 5014 scope.go:122] "RemoveContainer" containerID="09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb" Oct 30 18:19:21.479913 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.479898 5014 scope.go:122] "RemoveContainer" containerID="09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb" Oct 30 18:19:21.480148 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:19:21.480124 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb\": container with ID starting with 09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb not found: ID does not exist" containerID="09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb" Oct 30 18:19:21.480198 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.480158 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb"} err="failed to get container status \"09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb\": rpc error: code = NotFound desc = could not find container \"09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb\": container with ID starting with 09d3389c90554e6b7199b5b421f21c6f2452df31860f3fd7ca806a3e8fef7adb not found: ID does not exist" Oct 30 18:19:21.519688 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.519668 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:21.531073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.531045 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:21.561161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.561142 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:21.561511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.561485 5014 cpu_manager.go:414] "containerMap: removing container" podUID="1513b218-4536-47d9-a12b-798ac631d4d0" containerName="otc-container" Oct 30 18:19:21.561511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.561499 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1513b218-4536-47d9-a12b-798ac631d4d0" containerName="otc-container" Oct 30 18:19:21.561644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.561580 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1513b218-4536-47d9-a12b-798ac631d4d0" containerName="otc-container" Oct 30 18:19:21.564553 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.564539 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.578935 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.578913 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:19:21.579029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.578940 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:19:21.579029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.578957 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:19:21.579029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.578969 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:19:21.579029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.578996 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:19:21.579029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.579007 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:19:21.591704 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.591682 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:21.639952 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.639932 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.640046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.639968 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.640046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.639988 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km655\" (UniqueName: \"kubernetes.io/projected/7f5f7291-77fe-4639-ac93-501780a5844f-kube-api-access-km655\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.640114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.640100 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.640166 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.640153 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f5f7291-77fe-4639-ac93-501780a5844f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.640222 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.640211 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.741107 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.741050 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.741107 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.741099 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f5f7291-77fe-4639-ac93-501780a5844f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.741239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.741220 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.741316 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.741301 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.741371 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.741357 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.741406 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.741393 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km655\" (UniqueName: \"kubernetes.io/projected/7f5f7291-77fe-4639-ac93-501780a5844f-kube-api-access-km655\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.741891 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.741824 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f5f7291-77fe-4639-ac93-501780a5844f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.743770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.743746 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.743889 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.743835 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.743889 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.743845 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.743889 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.743873 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.763663 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.763641 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km655\" (UniqueName: \"kubernetes.io/projected/7f5f7291-77fe-4639-ac93-501780a5844f-kube-api-access-km655\") pod \"data-science-collector-collector-0\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:21.877189 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:21.877172 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:22.172456 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:22.172429 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:22.175754 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:19:22.175726 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5f7291_77fe_4639_ac93_501780a5844f.slice/crio-92ba327c488f72f935d10ca57a0dbdeb1c625bcbbc30fbbdcca1d4d694a8177a WatchSource:0}: Error finding container 92ba327c488f72f935d10ca57a0dbdeb1c625bcbbc30fbbdcca1d4d694a8177a: Status 404 returned error can't find the container with id 92ba327c488f72f935d10ca57a0dbdeb1c625bcbbc30fbbdcca1d4d694a8177a Oct 30 18:19:22.177925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:22.177895 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:19:22.178006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:22.177952 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:19:22.479296 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:22.479220 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f5f7291-77fe-4639-ac93-501780a5844f","Type":"ContainerStarted","Data":"2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405"} Oct 30 18:19:22.479296 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:22.479275 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f5f7291-77fe-4639-ac93-501780a5844f","Type":"ContainerStarted","Data":"92ba327c488f72f935d10ca57a0dbdeb1c625bcbbc30fbbdcca1d4d694a8177a"} Oct 30 18:19:22.521988 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:22.521931 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.5219141 podStartE2EDuration="1.5219141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:22.520913024 +0000 UTC m=+1375.509253512" watchObservedRunningTime="2026-10-30 18:19:22.5219141 +0000 UTC m=+1375.510254589" Oct 30 18:19:23.141688 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:23.141656 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1513b218-4536-47d9-a12b-798ac631d4d0" path="/var/lib/kubelet/pods/1513b218-4536-47d9-a12b-798ac631d4d0/volumes" Oct 30 18:19:34.522347 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.522280 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:19:34.529188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.529166 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.543587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.543559 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:19:34.543715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.543635 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:19:34.554929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.554908 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-gc99w\"" Oct 30 18:19:34.555052 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.554945 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:19:34.556758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.556736 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:19:34.642428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642402 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.642548 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642440 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.642548 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642461 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.642714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642565 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:19:34.642714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642564 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.642714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642637 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.642714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642672 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:19:34.642714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642699 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:19:34.642714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642698 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn55t\" (UniqueName: \"kubernetes.io/projected/47ea4956-38fb-4db5-8820-b27a952d2a79-kube-api-access-kn55t\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.642973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642717 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:19:34.642973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642770 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.642973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.642830 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.744194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.744168 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.744347 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.744202 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.744347 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.744253 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.744476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.744363 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.744476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.744392 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.744476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.744428 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.744685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.744633 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.744748 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.744695 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn55t\" (UniqueName: \"kubernetes.io/projected/47ea4956-38fb-4db5-8820-b27a952d2a79-kube-api-access-kn55t\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.745072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.745048 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.745393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.745112 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.745393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.745127 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.745541 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.745223 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.747044 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.747025 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.747161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.747144 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.747271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.747255 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:34.767880 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:34.767861 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn55t\" (UniqueName: \"kubernetes.io/projected/47ea4956-38fb-4db5-8820-b27a952d2a79-kube-api-access-kn55t\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:36.714982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:36.714952 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:36.715487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:36.715303 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="7f5f7291-77fe-4639-ac93-501780a5844f" containerName="otc-container" containerID="cri-o://2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405" gracePeriod=30 Oct 30 18:19:36.964228 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:36.964200 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:36.991856 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:36.991790 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:37.065532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:37.065505 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:37.065686 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:37.065667 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:37.079606 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:37.079573 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:37.179386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:37.179362 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:37.206743 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:37.206724 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:37.245860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:37.245809 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:37.567202 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:37.567170 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:19:37.580621 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:19:37.580581 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ea4956_38fb_4db5_8820_b27a952d2a79.slice/crio-360b603f7efcc9f452b131a2b404e3ad7688467361c87316f7273b3b6ec58a8b WatchSource:0}: Error finding container 360b603f7efcc9f452b131a2b404e3ad7688467361c87316f7273b3b6ec58a8b: Status 404 returned error can't find the container with id 360b603f7efcc9f452b131a2b404e3ad7688467361c87316f7273b3b6ec58a8b Oct 30 18:19:38.557403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:38.557366 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"47ea4956-38fb-4db5-8820-b27a952d2a79","Type":"ContainerStarted","Data":"887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb"} Oct 30 18:19:38.557863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:38.557424 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"47ea4956-38fb-4db5-8820-b27a952d2a79","Type":"ContainerStarted","Data":"ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8"} Oct 30 18:19:38.557863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:38.557448 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:38.557863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:38.557466 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:38.557863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:38.557479 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"47ea4956-38fb-4db5-8820-b27a952d2a79","Type":"ContainerStarted","Data":"adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89"} Oct 30 18:19:38.557863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:38.557495 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"47ea4956-38fb-4db5-8820-b27a952d2a79","Type":"ContainerStarted","Data":"360b603f7efcc9f452b131a2b404e3ad7688467361c87316f7273b3b6ec58a8b"} Oct 30 18:19:38.559914 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:38.559893 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:38.565873 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:38.565851 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:38.630726 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:38.630671 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=4.630654172 podStartE2EDuration="4.630654172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:38.618693492 +0000 UTC m=+1391.607034008" watchObservedRunningTime="2026-10-30 18:19:38.630654172 +0000 UTC m=+1391.618994659" Oct 30 18:19:39.074869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.074849 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.083844 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.083818 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-monitoring-tls-certs\") pod \"7f5f7291-77fe-4639-ac93-501780a5844f\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " Oct 30 18:19:39.083930 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.083891 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-data-science-collector-ta-client-cert\") pod \"7f5f7291-77fe-4639-ac93-501780a5844f\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " Oct 30 18:19:39.083987 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.083940 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5f7291-77fe-4639-ac93-501780a5844f-kube-api-access-km655\" (UniqueName: \"kubernetes.io/projected/7f5f7291-77fe-4639-ac93-501780a5844f-kube-api-access-km655\") pod \"7f5f7291-77fe-4639-ac93-501780a5844f\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " Oct 30 18:19:39.084047 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.084037 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-otlp-tls-certs\") pod \"7f5f7291-77fe-4639-ac93-501780a5844f\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " Oct 30 18:19:39.084104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.084070 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5f7291-77fe-4639-ac93-501780a5844f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f5f7291-77fe-4639-ac93-501780a5844f-otc-internal\") pod \"7f5f7291-77fe-4639-ac93-501780a5844f\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " Oct 30 18:19:39.084157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.084109 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-tls-certs\") pod \"7f5f7291-77fe-4639-ac93-501780a5844f\" (UID: \"7f5f7291-77fe-4639-ac93-501780a5844f\") " Oct 30 18:19:39.084480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.084450 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5f7291-77fe-4639-ac93-501780a5844f-otc-internal" pod "7f5f7291-77fe-4639-ac93-501780a5844f" (UID: "7f5f7291-77fe-4639-ac93-501780a5844f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:39.086245 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.086218 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-data-science-collector-ta-client-cert" pod "7f5f7291-77fe-4639-ac93-501780a5844f" (UID: "7f5f7291-77fe-4639-ac93-501780a5844f"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:39.086245 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.086238 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5f7291-77fe-4639-ac93-501780a5844f-kube-api-access-km655" pod "7f5f7291-77fe-4639-ac93-501780a5844f" (UID: "7f5f7291-77fe-4639-ac93-501780a5844f"). InnerVolumeSpecName "kube-api-access-km655". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:39.086633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.086581 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-monitoring-tls-certs" pod "7f5f7291-77fe-4639-ac93-501780a5844f" (UID: "7f5f7291-77fe-4639-ac93-501780a5844f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:39.086633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.086612 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-otlp-tls-certs" pod "7f5f7291-77fe-4639-ac93-501780a5844f" (UID: "7f5f7291-77fe-4639-ac93-501780a5844f"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:39.086790 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.086756 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-tls-certs" pod "7f5f7291-77fe-4639-ac93-501780a5844f" (UID: "7f5f7291-77fe-4639-ac93-501780a5844f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:39.185465 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.185435 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:39.185465 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.185466 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f5f7291-77fe-4639-ac93-501780a5844f-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:39.185640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.185494 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:39.185640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.185502 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:39.185640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.185513 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7f5f7291-77fe-4639-ac93-501780a5844f-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:39.185640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.185523 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km655\" (UniqueName: \"kubernetes.io/projected/7f5f7291-77fe-4639-ac93-501780a5844f-kube-api-access-km655\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:39.563496 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.563467 5014 generic.go:356] "Generic (PLEG): container finished" podID="7f5f7291-77fe-4639-ac93-501780a5844f" containerID="2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405" exitCode=0 Oct 30 18:19:39.563917 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.563528 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.563917 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.563552 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f5f7291-77fe-4639-ac93-501780a5844f","Type":"ContainerDied","Data":"2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405"} Oct 30 18:19:39.563917 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.563618 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f5f7291-77fe-4639-ac93-501780a5844f","Type":"ContainerDied","Data":"92ba327c488f72f935d10ca57a0dbdeb1c625bcbbc30fbbdcca1d4d694a8177a"} Oct 30 18:19:39.563917 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.563642 5014 scope.go:122] "RemoveContainer" containerID="2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405" Oct 30 18:19:39.563917 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.563910 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:39.572218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.572199 5014 scope.go:122] "RemoveContainer" containerID="2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405" Oct 30 18:19:39.572482 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:19:39.572464 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405\": container with ID starting with 2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405 not found: ID does not exist" containerID="2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405" Oct 30 18:19:39.572535 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.572491 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405"} err="failed to get container status \"2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405\": rpc error: code = NotFound desc = could not find container \"2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405\": container with ID starting with 2d08d9f9be47e4946b6c091d7d88263d02a561ecd67f55bd120ba22f0c011405 not found: ID does not exist" Oct 30 18:19:39.613164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.613140 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:39.624286 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.624265 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:39.655192 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.655171 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:39.655583 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.655565 5014 cpu_manager.go:414] "containerMap: removing container" podUID="7f5f7291-77fe-4639-ac93-501780a5844f" containerName="otc-container" Oct 30 18:19:39.655683 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.655584 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5f7291-77fe-4639-ac93-501780a5844f" containerName="otc-container" Oct 30 18:19:39.655774 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.655760 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5f7291-77fe-4639-ac93-501780a5844f" containerName="otc-container" Oct 30 18:19:39.667147 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.667131 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.681556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.681382 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:39.681680 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.681652 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:19:39.681934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.681837 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:19:39.681934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.681838 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:19:39.681934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.681862 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:19:39.681934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.681859 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:19:39.681934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.681906 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-50fd48e3\"" Oct 30 18:19:39.790270 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.790244 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.790413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.790293 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.790413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.790316 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72tkw\" (UniqueName: \"kubernetes.io/projected/ab08238e-3ca6-476c-bcfa-ed51814a60c4-kube-api-access-72tkw\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.790538 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.790416 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.790538 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.790481 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.790628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.790559 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.891281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.891217 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.891281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.891249 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72tkw\" (UniqueName: \"kubernetes.io/projected/ab08238e-3ca6-476c-bcfa-ed51814a60c4-kube-api-access-72tkw\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.891281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.891275 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.891519 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.891296 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.891519 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.891316 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.891519 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.891373 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.892397 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.892368 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.894372 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.894347 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.894486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.894470 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.894784 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.894765 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.894856 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.894765 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.913721 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.913703 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72tkw\" (UniqueName: \"kubernetes.io/projected/ab08238e-3ca6-476c-bcfa-ed51814a60c4-kube-api-access-72tkw\") pod \"data-science-collector-collector-0\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:39.979545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:39.979518 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:40.275028 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:40.274935 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:40.277277 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:19:40.277248 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab08238e_3ca6_476c_bcfa_ed51814a60c4.slice/crio-a653fb83877e9112b93e6eaec240cd846585b256ff643d0acd06ea34cfd9464b WatchSource:0}: Error finding container a653fb83877e9112b93e6eaec240cd846585b256ff643d0acd06ea34cfd9464b: Status 404 returned error can't find the container with id a653fb83877e9112b93e6eaec240cd846585b256ff643d0acd06ea34cfd9464b Oct 30 18:19:40.279569 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:40.279539 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:19:40.279683 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:40.279614 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:19:40.571053 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:40.571017 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ab08238e-3ca6-476c-bcfa-ed51814a60c4","Type":"ContainerStarted","Data":"b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136"} Oct 30 18:19:40.571510 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:40.571083 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ab08238e-3ca6-476c-bcfa-ed51814a60c4","Type":"ContainerStarted","Data":"a653fb83877e9112b93e6eaec240cd846585b256ff643d0acd06ea34cfd9464b"} Oct 30 18:19:40.613245 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:40.613201 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.61318805 podStartE2EDuration="1.61318805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:40.612270403 +0000 UTC m=+1393.600610891" watchObservedRunningTime="2026-10-30 18:19:40.61318805 +0000 UTC m=+1393.601528537" Oct 30 18:19:41.141863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:41.141830 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5f7291-77fe-4639-ac93-501780a5844f" path="/var/lib/kubelet/pods/7f5f7291-77fe-4639-ac93-501780a5844f/volumes" Oct 30 18:19:43.332678 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:19:43.332648 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:19:51.860496 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:51.860463 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:19:51.861063 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:51.860806 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo" containerID="cri-o://adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89" gracePeriod=30 Oct 30 18:19:51.861063 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:51.860972 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo-gateway-opa" containerID="cri-o://887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb" gracePeriod=30 Oct 30 18:19:51.861143 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:51.861067 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo-gateway" containerID="cri-o://ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8" gracePeriod=30 Oct 30 18:19:52.629025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:52.628991 5014 generic.go:356] "Generic (PLEG): container finished" podID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerID="887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb" exitCode=0 Oct 30 18:19:52.629025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:52.629017 5014 generic.go:356] "Generic (PLEG): container finished" podID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerID="adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89" exitCode=0 Oct 30 18:19:52.629270 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:52.629068 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"47ea4956-38fb-4db5-8820-b27a952d2a79","Type":"ContainerDied","Data":"887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb"} Oct 30 18:19:52.629270 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:52.629160 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"47ea4956-38fb-4db5-8820-b27a952d2a79","Type":"ContainerDied","Data":"adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89"} Oct 30 18:19:53.128916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.128888 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:53.200560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.200495 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-mtls\") pod \"47ea4956-38fb-4db5-8820-b27a952d2a79\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " Oct 30 18:19:53.200560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.200526 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-tenants\") pod \"47ea4956-38fb-4db5-8820-b27a952d2a79\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " Oct 30 18:19:53.200738 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.200619 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"47ea4956-38fb-4db5-8820-b27a952d2a79\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " Oct 30 18:19:53.200738 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.200720 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"47ea4956-38fb-4db5-8820-b27a952d2a79\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " Oct 30 18:19:53.200821 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.200749 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-ca-bundle\") pod \"47ea4956-38fb-4db5-8820-b27a952d2a79\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " Oct 30 18:19:53.200821 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.200808 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-rbac\") pod \"47ea4956-38fb-4db5-8820-b27a952d2a79\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " Oct 30 18:19:53.200920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.200834 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ea4956-38fb-4db5-8820-b27a952d2a79-kube-api-access-kn55t\" (UniqueName: \"kubernetes.io/projected/47ea4956-38fb-4db5-8820-b27a952d2a79-kube-api-access-kn55t\") pod \"47ea4956-38fb-4db5-8820-b27a952d2a79\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " Oct 30 18:19:53.200920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.200867 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"47ea4956-38fb-4db5-8820-b27a952d2a79\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " Oct 30 18:19:53.200920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.200893 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-conf\") pod \"47ea4956-38fb-4db5-8820-b27a952d2a79\" (UID: \"47ea4956-38fb-4db5-8820-b27a952d2a79\") " Oct 30 18:19:53.201516 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.201238 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-ca-bundle" pod "47ea4956-38fb-4db5-8820-b27a952d2a79" (UID: "47ea4956-38fb-4db5-8820-b27a952d2a79"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:53.201516 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.201262 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-rbac" pod "47ea4956-38fb-4db5-8820-b27a952d2a79" (UID: "47ea4956-38fb-4db5-8820-b27a952d2a79"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:53.201516 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.201474 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-conf" pod "47ea4956-38fb-4db5-8820-b27a952d2a79" (UID: "47ea4956-38fb-4db5-8820-b27a952d2a79"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:53.201934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.201880 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-serving-cabundle" pod "47ea4956-38fb-4db5-8820-b27a952d2a79" (UID: "47ea4956-38fb-4db5-8820-b27a952d2a79"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:53.203320 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.203295 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-mtls" pod "47ea4956-38fb-4db5-8820-b27a952d2a79" (UID: "47ea4956-38fb-4db5-8820-b27a952d2a79"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:53.203588 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.203568 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-tenants" pod "47ea4956-38fb-4db5-8820-b27a952d2a79" (UID: "47ea4956-38fb-4db5-8820-b27a952d2a79"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:53.203588 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.203649 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ea4956-38fb-4db5-8820-b27a952d2a79-kube-api-access-kn55t" pod "47ea4956-38fb-4db5-8820-b27a952d2a79" (UID: "47ea4956-38fb-4db5-8820-b27a952d2a79"). InnerVolumeSpecName "kube-api-access-kn55t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:53.204082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.204054 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "47ea4956-38fb-4db5-8820-b27a952d2a79" (UID: "47ea4956-38fb-4db5-8820-b27a952d2a79"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:53.204237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.204218 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "47ea4956-38fb-4db5-8820-b27a952d2a79" (UID: "47ea4956-38fb-4db5-8820-b27a952d2a79"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:19:53.301899 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.301867 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:53.302017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.301918 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:19:53.302017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.301937 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:53.302017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.301951 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:53.302017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.301966 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn55t\" (UniqueName: \"kubernetes.io/projected/47ea4956-38fb-4db5-8820-b27a952d2a79-kube-api-access-kn55t\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:53.302017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.301981 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:53.302017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.301996 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:53.302017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.302010 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:53.302430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.302023 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/47ea4956-38fb-4db5-8820-b27a952d2a79-gateway-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:53.311191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.311174 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:19:53.403229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.403208 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" Oct 30 18:19:53.635883 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.635859 5014 generic.go:356] "Generic (PLEG): container finished" podID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerID="ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8" exitCode=0 Oct 30 18:19:53.636028 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.635918 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:19:53.636028 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.635922 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"47ea4956-38fb-4db5-8820-b27a952d2a79","Type":"ContainerDied","Data":"ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8"} Oct 30 18:19:53.636028 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.635998 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"47ea4956-38fb-4db5-8820-b27a952d2a79","Type":"ContainerDied","Data":"360b603f7efcc9f452b131a2b404e3ad7688467361c87316f7273b3b6ec58a8b"} Oct 30 18:19:53.636188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.636037 5014 scope.go:122] "RemoveContainer" containerID="887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb" Oct 30 18:19:53.645433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.645414 5014 scope.go:122] "RemoveContainer" containerID="ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8" Oct 30 18:19:53.653194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.653177 5014 scope.go:122] "RemoveContainer" containerID="adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89" Oct 30 18:19:53.660624 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.660604 5014 scope.go:122] "RemoveContainer" containerID="887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb" Oct 30 18:19:53.660891 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:19:53.660874 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb\": container with ID starting with 887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb not found: ID does not exist" containerID="887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb" Oct 30 18:19:53.660954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.660900 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb"} err="failed to get container status \"887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb\": rpc error: code = NotFound desc = could not find container \"887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb\": container with ID starting with 887c73cfd416fbdbeae58106e67ec3fbcd2ca108756516ae0f8b480b9a006edb not found: ID does not exist" Oct 30 18:19:53.660954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.660917 5014 scope.go:122] "RemoveContainer" containerID="ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8" Oct 30 18:19:53.661139 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:19:53.661124 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8\": container with ID starting with ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8 not found: ID does not exist" containerID="ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8" Oct 30 18:19:53.661177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.661143 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8"} err="failed to get container status \"ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8\": rpc error: code = NotFound desc = could not find container \"ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8\": container with ID starting with ed23fd696ddcf01f6d26be4da0aa5a0b7eddd4f565a492f56be8d83cfe37d2b8 not found: ID does not exist" Oct 30 18:19:53.661177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.661156 5014 scope.go:122] "RemoveContainer" containerID="adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89" Oct 30 18:19:53.661354 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:19:53.661339 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89\": container with ID starting with adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89 not found: ID does not exist" containerID="adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89" Oct 30 18:19:53.661395 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.661370 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89"} err="failed to get container status \"adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89\": rpc error: code = NotFound desc = could not find container \"adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89\": container with ID starting with adc259f2e45dd83f710c93ec428ac677c24afc467140c3a73eec050ac874bf89 not found: ID does not exist" Oct 30 18:19:53.700144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.700124 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:19:53.706025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.706007 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:19:53.828618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.828576 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:53.828844 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:53.828820 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="ab08238e-3ca6-476c-bcfa-ed51814a60c4" containerName="otc-container" containerID="cri-o://b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136" gracePeriod=30 Oct 30 18:19:55.140991 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:55.140959 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" path="/var/lib/kubelet/pods/47ea4956-38fb-4db5-8820-b27a952d2a79/volumes" Oct 30 18:19:56.182732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.182710 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.323047 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.323021 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-monitoring-tls-certs\") pod \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " Oct 30 18:19:56.323171 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.323077 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab08238e-3ca6-476c-bcfa-ed51814a60c4-kube-api-access-72tkw\" (UniqueName: \"kubernetes.io/projected/ab08238e-3ca6-476c-bcfa-ed51814a60c4-kube-api-access-72tkw\") pod \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " Oct 30 18:19:56.323171 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.323120 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otc-internal\") pod \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " Oct 30 18:19:56.323171 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.323147 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-data-science-collector-ta-client-cert\") pod \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " Oct 30 18:19:56.323307 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.323190 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otlp-tls-certs\") pod \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " Oct 30 18:19:56.323307 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.323237 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-tls-certs\") pod \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\" (UID: \"ab08238e-3ca6-476c-bcfa-ed51814a60c4\") " Oct 30 18:19:56.323549 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.323521 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otc-internal" pod "ab08238e-3ca6-476c-bcfa-ed51814a60c4" (UID: "ab08238e-3ca6-476c-bcfa-ed51814a60c4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:56.325361 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.325336 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab08238e-3ca6-476c-bcfa-ed51814a60c4-kube-api-access-72tkw" pod "ab08238e-3ca6-476c-bcfa-ed51814a60c4" (UID: "ab08238e-3ca6-476c-bcfa-ed51814a60c4"). InnerVolumeSpecName "kube-api-access-72tkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:56.325463 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.325340 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-monitoring-tls-certs" pod "ab08238e-3ca6-476c-bcfa-ed51814a60c4" (UID: "ab08238e-3ca6-476c-bcfa-ed51814a60c4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:56.325554 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.325529 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-tls-certs" pod "ab08238e-3ca6-476c-bcfa-ed51814a60c4" (UID: "ab08238e-3ca6-476c-bcfa-ed51814a60c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:56.325920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.325897 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-data-science-collector-ta-client-cert" pod "ab08238e-3ca6-476c-bcfa-ed51814a60c4" (UID: "ab08238e-3ca6-476c-bcfa-ed51814a60c4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:56.325972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.325918 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otlp-tls-certs" pod "ab08238e-3ca6-476c-bcfa-ed51814a60c4" (UID: "ab08238e-3ca6-476c-bcfa-ed51814a60c4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:56.424554 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.424529 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:56.424554 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.424549 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72tkw\" (UniqueName: \"kubernetes.io/projected/ab08238e-3ca6-476c-bcfa-ed51814a60c4-kube-api-access-72tkw\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:56.424694 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.424560 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:56.424694 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.424570 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:56.424694 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.424580 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:56.424694 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.424588 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab08238e-3ca6-476c-bcfa-ed51814a60c4-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:19:56.651499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.651436 5014 generic.go:356] "Generic (PLEG): container finished" podID="ab08238e-3ca6-476c-bcfa-ed51814a60c4" containerID="b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136" exitCode=0 Oct 30 18:19:56.651499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.651486 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ab08238e-3ca6-476c-bcfa-ed51814a60c4","Type":"ContainerDied","Data":"b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136"} Oct 30 18:19:56.651499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.651489 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.651744 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.651527 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ab08238e-3ca6-476c-bcfa-ed51814a60c4","Type":"ContainerDied","Data":"a653fb83877e9112b93e6eaec240cd846585b256ff643d0acd06ea34cfd9464b"} Oct 30 18:19:56.651744 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.651547 5014 scope.go:122] "RemoveContainer" containerID="b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136" Oct 30 18:19:56.660782 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.660765 5014 scope.go:122] "RemoveContainer" containerID="b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136" Oct 30 18:19:56.661093 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:19:56.661074 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136\": container with ID starting with b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136 not found: ID does not exist" containerID="b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136" Oct 30 18:19:56.661152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.661099 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136"} err="failed to get container status \"b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136\": rpc error: code = NotFound desc = could not find container \"b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136\": container with ID starting with b33b41b588e65e8287d2df2f645a221e02dd1ca2df1c838f5e7b115cd92ca136 not found: ID does not exist" Oct 30 18:19:56.701122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.701097 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:56.705554 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.705536 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:56.743440 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743417 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:56.743829 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743807 5014 cpu_manager.go:414] "containerMap: removing container" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo" Oct 30 18:19:56.743829 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743827 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo" Oct 30 18:19:56.743962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743843 5014 cpu_manager.go:414] "containerMap: removing container" podUID="ab08238e-3ca6-476c-bcfa-ed51814a60c4" containerName="otc-container" Oct 30 18:19:56.743962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743848 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab08238e-3ca6-476c-bcfa-ed51814a60c4" containerName="otc-container" Oct 30 18:19:56.743962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743860 5014 cpu_manager.go:414] "containerMap: removing container" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo-gateway" Oct 30 18:19:56.743962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743864 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo-gateway" Oct 30 18:19:56.743962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743881 5014 cpu_manager.go:414] "containerMap: removing container" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo-gateway-opa" Oct 30 18:19:56.743962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743886 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo-gateway-opa" Oct 30 18:19:56.743962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743947 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo" Oct 30 18:19:56.743962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743955 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab08238e-3ca6-476c-bcfa-ed51814a60c4" containerName="otc-container" Oct 30 18:19:56.743962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743963 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo-gateway" Oct 30 18:19:56.744216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.743971 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ea4956-38fb-4db5-8820-b27a952d2a79" containerName="tempo-gateway-opa" Oct 30 18:19:56.748581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.748566 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.763399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.763384 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:19:56.763479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.763432 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:19:56.763479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.763461 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:19:56.763574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.763563 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:19:56.766109 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.766070 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:19:56.768086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.768063 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:56.826787 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.826764 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9c410df7-2fad-4f98-92f3-04eca9436344-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.826920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.826798 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.826920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.826829 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.826920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.826911 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.827059 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.826970 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dfc9\" (UniqueName: \"kubernetes.io/projected/9c410df7-2fad-4f98-92f3-04eca9436344-kube-api-access-5dfc9\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.827112 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.827083 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.863843 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.863822 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:19:56.927881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.927831 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.927881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.927875 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9c410df7-2fad-4f98-92f3-04eca9436344-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.928029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.927905 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.928029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.927933 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.928029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.927967 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.928176 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.928099 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dfc9\" (UniqueName: \"kubernetes.io/projected/9c410df7-2fad-4f98-92f3-04eca9436344-kube-api-access-5dfc9\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.928718 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.928699 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9c410df7-2fad-4f98-92f3-04eca9436344-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.930574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.930555 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.930574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.930570 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.931059 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.931040 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.931351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.931336 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:56.950362 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:56.950344 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dfc9\" (UniqueName: \"kubernetes.io/projected/9c410df7-2fad-4f98-92f3-04eca9436344-kube-api-access-5dfc9\") pod \"data-science-collector-collector-0\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:57.060684 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:57.060663 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:19:57.144229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:57.144201 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab08238e-3ca6-476c-bcfa-ed51814a60c4" path="/var/lib/kubelet/pods/ab08238e-3ca6-476c-bcfa-ed51814a60c4/volumes" Oct 30 18:19:57.358567 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:57.358547 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:19:57.360570 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:19:57.360537 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c410df7_2fad_4f98_92f3_04eca9436344.slice/crio-2fce6908aa5ae1e1fbe324a8b1d6800d3550d54da809269c92afdd0321b5ff89 WatchSource:0}: Error finding container 2fce6908aa5ae1e1fbe324a8b1d6800d3550d54da809269c92afdd0321b5ff89: Status 404 returned error can't find the container with id 2fce6908aa5ae1e1fbe324a8b1d6800d3550d54da809269c92afdd0321b5ff89 Oct 30 18:19:57.362889 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:57.362861 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:19:57.362965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:57.362915 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:19:57.658580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:57.658497 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9c410df7-2fad-4f98-92f3-04eca9436344","Type":"ContainerStarted","Data":"51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c"} Oct 30 18:19:57.658580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:57.658578 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9c410df7-2fad-4f98-92f3-04eca9436344","Type":"ContainerStarted","Data":"2fce6908aa5ae1e1fbe324a8b1d6800d3550d54da809269c92afdd0321b5ff89"} Oct 30 18:19:57.701587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:19:57.701540 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.701526259 podStartE2EDuration="1.701526259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:57.699821136 +0000 UTC m=+1410.688161623" watchObservedRunningTime="2026-10-30 18:19:57.701526259 +0000 UTC m=+1410.689866752" Oct 30 18:20:09.925022 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:09.924990 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:09.925776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:09.925684 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="9c410df7-2fad-4f98-92f3-04eca9436344" containerName="otc-container" containerID="cri-o://51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c" gracePeriod=30 Oct 30 18:20:10.170575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.170550 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:10.230747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.230686 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-data-science-collector-ta-client-cert\") pod \"9c410df7-2fad-4f98-92f3-04eca9436344\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " Oct 30 18:20:10.230884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.230755 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-otlp-tls-certs\") pod \"9c410df7-2fad-4f98-92f3-04eca9436344\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " Oct 30 18:20:10.230948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.230873 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c410df7-2fad-4f98-92f3-04eca9436344-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9c410df7-2fad-4f98-92f3-04eca9436344-otc-internal\") pod \"9c410df7-2fad-4f98-92f3-04eca9436344\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " Oct 30 18:20:10.231006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.230977 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c410df7-2fad-4f98-92f3-04eca9436344-kube-api-access-5dfc9\" (UniqueName: \"kubernetes.io/projected/9c410df7-2fad-4f98-92f3-04eca9436344-kube-api-access-5dfc9\") pod \"9c410df7-2fad-4f98-92f3-04eca9436344\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " Oct 30 18:20:10.231061 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.231015 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-tls-certs\") pod \"9c410df7-2fad-4f98-92f3-04eca9436344\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " Oct 30 18:20:10.231207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.231182 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c410df7-2fad-4f98-92f3-04eca9436344-otc-internal" pod "9c410df7-2fad-4f98-92f3-04eca9436344" (UID: "9c410df7-2fad-4f98-92f3-04eca9436344"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:10.231618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.231365 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9c410df7-2fad-4f98-92f3-04eca9436344-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:10.233287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.233258 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c410df7-2fad-4f98-92f3-04eca9436344-kube-api-access-5dfc9" pod "9c410df7-2fad-4f98-92f3-04eca9436344" (UID: "9c410df7-2fad-4f98-92f3-04eca9436344"). InnerVolumeSpecName "kube-api-access-5dfc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:10.233398 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.233314 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-otlp-tls-certs" pod "9c410df7-2fad-4f98-92f3-04eca9436344" (UID: "9c410df7-2fad-4f98-92f3-04eca9436344"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:10.233398 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.233341 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-tls-certs" pod "9c410df7-2fad-4f98-92f3-04eca9436344" (UID: "9c410df7-2fad-4f98-92f3-04eca9436344"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:10.233512 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.233491 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-data-science-collector-ta-client-cert" pod "9c410df7-2fad-4f98-92f3-04eca9436344" (UID: "9c410df7-2fad-4f98-92f3-04eca9436344"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:10.332338 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.332312 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-monitoring-tls-certs\") pod \"9c410df7-2fad-4f98-92f3-04eca9436344\" (UID: \"9c410df7-2fad-4f98-92f3-04eca9436344\") " Oct 30 18:20:10.332541 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.332525 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:10.332638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.332546 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:10.332638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.332562 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dfc9\" (UniqueName: \"kubernetes.io/projected/9c410df7-2fad-4f98-92f3-04eca9436344-kube-api-access-5dfc9\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:10.332638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.332575 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:10.334092 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.334073 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-monitoring-tls-certs" pod "9c410df7-2fad-4f98-92f3-04eca9436344" (UID: "9c410df7-2fad-4f98-92f3-04eca9436344"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:10.433823 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.433802 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c410df7-2fad-4f98-92f3-04eca9436344-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:10.721908 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.721883 5014 generic.go:356] "Generic (PLEG): container finished" podID="9c410df7-2fad-4f98-92f3-04eca9436344" containerID="51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c" exitCode=0 Oct 30 18:20:10.722014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.721932 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:10.722014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.721967 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9c410df7-2fad-4f98-92f3-04eca9436344","Type":"ContainerDied","Data":"51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c"} Oct 30 18:20:10.722097 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.722028 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9c410df7-2fad-4f98-92f3-04eca9436344","Type":"ContainerDied","Data":"2fce6908aa5ae1e1fbe324a8b1d6800d3550d54da809269c92afdd0321b5ff89"} Oct 30 18:20:10.722097 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.722053 5014 scope.go:122] "RemoveContainer" containerID="51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c" Oct 30 18:20:10.731770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.731753 5014 scope.go:122] "RemoveContainer" containerID="51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c" Oct 30 18:20:10.732047 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:20:10.732025 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c\": container with ID starting with 51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c not found: ID does not exist" containerID="51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c" Oct 30 18:20:10.732137 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.732054 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c"} err="failed to get container status \"51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c\": rpc error: code = NotFound desc = could not find container \"51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c\": container with ID starting with 51ac0fa8cee3367c72327b2419db1c82997f9a667e716957828480fe68178d3c not found: ID does not exist" Oct 30 18:20:10.771555 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.771531 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:10.783373 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.783353 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:10.813848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.813826 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:10.814249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.814232 5014 cpu_manager.go:414] "containerMap: removing container" podUID="9c410df7-2fad-4f98-92f3-04eca9436344" containerName="otc-container" Oct 30 18:20:10.814337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.814250 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c410df7-2fad-4f98-92f3-04eca9436344" containerName="otc-container" Oct 30 18:20:10.814389 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.814352 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c410df7-2fad-4f98-92f3-04eca9436344" containerName="otc-container" Oct 30 18:20:10.818862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.818846 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:10.833240 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.833222 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:20:10.833341 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.833323 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:20:10.833388 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.833339 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:20:10.833388 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.833363 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:20:10.833388 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.833364 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:20:10.833497 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.833384 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:20:10.838648 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.838630 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:10.938206 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.938175 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wt5r\" (UniqueName: \"kubernetes.io/projected/011cd0cb-2065-438b-b835-06c9cfc38c01-kube-api-access-9wt5r\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:10.938545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.938248 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:10.938545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.938327 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:10.938545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.938372 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:10.938545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.938453 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:10.938545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:10.938491 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/011cd0cb-2065-438b-b835-06c9cfc38c01-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.039579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.039518 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.039579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.039564 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.039759 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.039630 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.039759 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.039669 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/011cd0cb-2065-438b-b835-06c9cfc38c01-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.039863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.039788 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wt5r\" (UniqueName: \"kubernetes.io/projected/011cd0cb-2065-438b-b835-06c9cfc38c01-kube-api-access-9wt5r\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.039863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.039854 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.040726 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.040699 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/011cd0cb-2065-438b-b835-06c9cfc38c01-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.042199 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.042179 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.042286 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.042186 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.042369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.042350 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.042407 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.042357 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.064331 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.064300 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wt5r\" (UniqueName: \"kubernetes.io/projected/011cd0cb-2065-438b-b835-06c9cfc38c01-kube-api-access-9wt5r\") pod \"data-science-collector-collector-0\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.130395 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.130376 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:11.142129 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.142107 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c410df7-2fad-4f98-92f3-04eca9436344" path="/var/lib/kubelet/pods/9c410df7-2fad-4f98-92f3-04eca9436344/volumes" Oct 30 18:20:11.426764 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.426705 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:11.429685 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:20:11.429662 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011cd0cb_2065_438b_b835_06c9cfc38c01.slice/crio-a28b9bf16e70c00fc115a6fe372734f984de2aed7394aefdcc5d842127e2cf0b WatchSource:0}: Error finding container a28b9bf16e70c00fc115a6fe372734f984de2aed7394aefdcc5d842127e2cf0b: Status 404 returned error can't find the container with id a28b9bf16e70c00fc115a6fe372734f984de2aed7394aefdcc5d842127e2cf0b Oct 30 18:20:11.431711 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.431682 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:20:11.431781 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.431738 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:20:11.729906 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.729824 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"011cd0cb-2065-438b-b835-06c9cfc38c01","Type":"ContainerStarted","Data":"aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741"} Oct 30 18:20:11.729906 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.729871 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"011cd0cb-2065-438b-b835-06c9cfc38c01","Type":"ContainerStarted","Data":"a28b9bf16e70c00fc115a6fe372734f984de2aed7394aefdcc5d842127e2cf0b"} Oct 30 18:20:11.772015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:11.771961 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.77194491 podStartE2EDuration="1.77194491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:11.770934874 +0000 UTC m=+1424.759275364" watchObservedRunningTime="2026-10-30 18:20:11.77194491 +0000 UTC m=+1424.760285398" Oct 30 18:20:13.343228 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:20:13.343188 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:20:29.035540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:29.035502 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:29.036052 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:29.035776 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="011cd0cb-2065-438b-b835-06c9cfc38c01" containerName="otc-container" containerID="cri-o://aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741" gracePeriod=30 Oct 30 18:20:30.385428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.385405 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:30.490407 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.490344 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-data-science-collector-ta-client-cert\") pod \"011cd0cb-2065-438b-b835-06c9cfc38c01\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " Oct 30 18:20:30.490557 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.490431 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-monitoring-tls-certs\") pod \"011cd0cb-2065-438b-b835-06c9cfc38c01\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " Oct 30 18:20:30.490557 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.490467 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-otlp-tls-certs\") pod \"011cd0cb-2065-438b-b835-06c9cfc38c01\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " Oct 30 18:20:30.490557 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.490495 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/011cd0cb-2065-438b-b835-06c9cfc38c01-otc-internal\" (UniqueName: \"kubernetes.io/configmap/011cd0cb-2065-438b-b835-06c9cfc38c01-otc-internal\") pod \"011cd0cb-2065-438b-b835-06c9cfc38c01\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " Oct 30 18:20:30.490831 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.490641 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011cd0cb-2065-438b-b835-06c9cfc38c01-kube-api-access-9wt5r\" (UniqueName: \"kubernetes.io/projected/011cd0cb-2065-438b-b835-06c9cfc38c01-kube-api-access-9wt5r\") pod \"011cd0cb-2065-438b-b835-06c9cfc38c01\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " Oct 30 18:20:30.490831 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.490724 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-tls-certs\") pod \"011cd0cb-2065-438b-b835-06c9cfc38c01\" (UID: \"011cd0cb-2065-438b-b835-06c9cfc38c01\") " Oct 30 18:20:30.490939 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.490895 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011cd0cb-2065-438b-b835-06c9cfc38c01-otc-internal" pod "011cd0cb-2065-438b-b835-06c9cfc38c01" (UID: "011cd0cb-2065-438b-b835-06c9cfc38c01"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:30.491164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.491047 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/011cd0cb-2065-438b-b835-06c9cfc38c01-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:30.492945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.492914 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-otlp-tls-certs" pod "011cd0cb-2065-438b-b835-06c9cfc38c01" (UID: "011cd0cb-2065-438b-b835-06c9cfc38c01"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:30.493072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.493047 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-data-science-collector-ta-client-cert" pod "011cd0cb-2065-438b-b835-06c9cfc38c01" (UID: "011cd0cb-2065-438b-b835-06c9cfc38c01"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:30.493072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.493056 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011cd0cb-2065-438b-b835-06c9cfc38c01-kube-api-access-9wt5r" pod "011cd0cb-2065-438b-b835-06c9cfc38c01" (UID: "011cd0cb-2065-438b-b835-06c9cfc38c01"). InnerVolumeSpecName "kube-api-access-9wt5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:30.493245 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.493202 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-monitoring-tls-certs" pod "011cd0cb-2065-438b-b835-06c9cfc38c01" (UID: "011cd0cb-2065-438b-b835-06c9cfc38c01"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:30.493437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.493417 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-tls-certs" pod "011cd0cb-2065-438b-b835-06c9cfc38c01" (UID: "011cd0cb-2065-438b-b835-06c9cfc38c01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:30.592375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.592355 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:30.592375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.592375 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:30.592493 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.592384 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wt5r\" (UniqueName: \"kubernetes.io/projected/011cd0cb-2065-438b-b835-06c9cfc38c01-kube-api-access-9wt5r\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:30.592493 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.592393 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:30.592493 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.592405 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/011cd0cb-2065-438b-b835-06c9cfc38c01-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:30.830419 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.830392 5014 generic.go:356] "Generic (PLEG): container finished" podID="011cd0cb-2065-438b-b835-06c9cfc38c01" containerID="aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741" exitCode=0 Oct 30 18:20:30.830521 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.830444 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:30.830521 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.830461 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"011cd0cb-2065-438b-b835-06c9cfc38c01","Type":"ContainerDied","Data":"aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741"} Oct 30 18:20:30.830521 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.830511 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"011cd0cb-2065-438b-b835-06c9cfc38c01","Type":"ContainerDied","Data":"a28b9bf16e70c00fc115a6fe372734f984de2aed7394aefdcc5d842127e2cf0b"} Oct 30 18:20:30.830688 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.830540 5014 scope.go:122] "RemoveContainer" containerID="aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741" Oct 30 18:20:30.839796 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.839780 5014 scope.go:122] "RemoveContainer" containerID="aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741" Oct 30 18:20:30.840069 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:20:30.840050 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741\": container with ID starting with aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741 not found: ID does not exist" containerID="aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741" Oct 30 18:20:30.840116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.840077 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741"} err="failed to get container status \"aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741\": rpc error: code = NotFound desc = could not find container \"aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741\": container with ID starting with aa7a02fc52c6a49eb86b2c4d8aa2078ebfecd99f0f3a9a0c9e1a8e7175652741 not found: ID does not exist" Oct 30 18:20:30.884682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.881789 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:30.890989 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.890965 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:30.922136 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.922109 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:30.922513 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.922498 5014 cpu_manager.go:414] "containerMap: removing container" podUID="011cd0cb-2065-438b-b835-06c9cfc38c01" containerName="otc-container" Oct 30 18:20:30.922589 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.922516 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011cd0cb-2065-438b-b835-06c9cfc38c01" containerName="otc-container" Oct 30 18:20:30.922675 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.922648 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="011cd0cb-2065-438b-b835-06c9cfc38c01" containerName="otc-container" Oct 30 18:20:30.927072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.927054 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:30.941678 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.941657 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:20:30.941678 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.941674 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:20:30.941821 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.941673 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:20:30.941821 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.941675 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:20:30.941821 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.941701 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:20:30.941821 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.941778 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:20:30.947896 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:30.947867 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:31.097325 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.097251 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.097325 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.097312 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.097473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.097386 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6rm\" (UniqueName: \"kubernetes.io/projected/940b546a-101b-494a-bb85-ea4ba70c5b4f-kube-api-access-8m6rm\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.097473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.097424 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/940b546a-101b-494a-bb85-ea4ba70c5b4f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.097559 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.097526 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.097629 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.097559 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.141212 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.141177 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="011cd0cb-2065-438b-b835-06c9cfc38c01" path="/var/lib/kubelet/pods/011cd0cb-2065-438b-b835-06c9cfc38c01/volumes" Oct 30 18:20:31.198630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.198575 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.198630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.198635 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6rm\" (UniqueName: \"kubernetes.io/projected/940b546a-101b-494a-bb85-ea4ba70c5b4f-kube-api-access-8m6rm\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.198817 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.198654 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/940b546a-101b-494a-bb85-ea4ba70c5b4f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.198817 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.198686 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.198817 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.198707 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.198817 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.198774 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.199651 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.199616 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/940b546a-101b-494a-bb85-ea4ba70c5b4f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.201308 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.201279 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.201308 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.201301 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.201474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.201413 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.201474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.201432 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.221231 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.221204 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6rm\" (UniqueName: \"kubernetes.io/projected/940b546a-101b-494a-bb85-ea4ba70c5b4f-kube-api-access-8m6rm\") pod \"data-science-collector-collector-0\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.239020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.239000 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:31.546204 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.545970 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:31.548691 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:20:31.548667 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940b546a_101b_494a_bb85_ea4ba70c5b4f.slice/crio-4ffe41b4686c1e3e47d0e13e7055dd0fea61ee65199f24adc38b5beae3f38072 WatchSource:0}: Error finding container 4ffe41b4686c1e3e47d0e13e7055dd0fea61ee65199f24adc38b5beae3f38072: Status 404 returned error can't find the container with id 4ffe41b4686c1e3e47d0e13e7055dd0fea61ee65199f24adc38b5beae3f38072 Oct 30 18:20:31.550781 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.550752 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:20:31.550846 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.550810 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:20:31.837287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.837258 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"940b546a-101b-494a-bb85-ea4ba70c5b4f","Type":"ContainerStarted","Data":"101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07"} Oct 30 18:20:31.837475 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.837317 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"940b546a-101b-494a-bb85-ea4ba70c5b4f","Type":"ContainerStarted","Data":"4ffe41b4686c1e3e47d0e13e7055dd0fea61ee65199f24adc38b5beae3f38072"} Oct 30 18:20:31.880033 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:31.879989 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.879975975 podStartE2EDuration="1.879975975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:31.879037647 +0000 UTC m=+1444.867378136" watchObservedRunningTime="2026-10-30 18:20:31.879975975 +0000 UTC m=+1444.868316462" Oct 30 18:20:43.334916 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:20:43.334880 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:20:52.159003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:52.158948 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:52.159627 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:52.159255 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="940b546a-101b-494a-bb85-ea4ba70c5b4f" containerName="otc-container" containerID="cri-o://101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07" gracePeriod=30 Oct 30 18:20:54.417682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.417661 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:54.471226 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.471197 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-monitoring-tls-certs\") pod \"940b546a-101b-494a-bb85-ea4ba70c5b4f\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " Oct 30 18:20:54.471394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.471246 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940b546a-101b-494a-bb85-ea4ba70c5b4f-kube-api-access-8m6rm\" (UniqueName: \"kubernetes.io/projected/940b546a-101b-494a-bb85-ea4ba70c5b4f-kube-api-access-8m6rm\") pod \"940b546a-101b-494a-bb85-ea4ba70c5b4f\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " Oct 30 18:20:54.471394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.471290 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-data-science-collector-ta-client-cert\") pod \"940b546a-101b-494a-bb85-ea4ba70c5b4f\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " Oct 30 18:20:54.471394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.471319 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-otlp-tls-certs\") pod \"940b546a-101b-494a-bb85-ea4ba70c5b4f\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " Oct 30 18:20:54.471394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.471357 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-tls-certs\") pod \"940b546a-101b-494a-bb85-ea4ba70c5b4f\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " Oct 30 18:20:54.471394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.471384 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940b546a-101b-494a-bb85-ea4ba70c5b4f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/940b546a-101b-494a-bb85-ea4ba70c5b4f-otc-internal\") pod \"940b546a-101b-494a-bb85-ea4ba70c5b4f\" (UID: \"940b546a-101b-494a-bb85-ea4ba70c5b4f\") " Oct 30 18:20:54.472339 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.472140 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940b546a-101b-494a-bb85-ea4ba70c5b4f-otc-internal" pod "940b546a-101b-494a-bb85-ea4ba70c5b4f" (UID: "940b546a-101b-494a-bb85-ea4ba70c5b4f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:54.475728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.474275 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-data-science-collector-ta-client-cert" pod "940b546a-101b-494a-bb85-ea4ba70c5b4f" (UID: "940b546a-101b-494a-bb85-ea4ba70c5b4f"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:54.475728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.474483 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-tls-certs" pod "940b546a-101b-494a-bb85-ea4ba70c5b4f" (UID: "940b546a-101b-494a-bb85-ea4ba70c5b4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:54.475728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.474679 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-otlp-tls-certs" pod "940b546a-101b-494a-bb85-ea4ba70c5b4f" (UID: "940b546a-101b-494a-bb85-ea4ba70c5b4f"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:54.475728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.474769 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-monitoring-tls-certs" pod "940b546a-101b-494a-bb85-ea4ba70c5b4f" (UID: "940b546a-101b-494a-bb85-ea4ba70c5b4f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:54.475728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.474915 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940b546a-101b-494a-bb85-ea4ba70c5b4f-kube-api-access-8m6rm" pod "940b546a-101b-494a-bb85-ea4ba70c5b4f" (UID: "940b546a-101b-494a-bb85-ea4ba70c5b4f"). InnerVolumeSpecName "kube-api-access-8m6rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:54.572174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.572154 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:54.572256 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.572176 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6rm\" (UniqueName: \"kubernetes.io/projected/940b546a-101b-494a-bb85-ea4ba70c5b4f-kube-api-access-8m6rm\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:54.572256 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.572189 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:54.572256 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.572198 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:54.572256 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.572206 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940b546a-101b-494a-bb85-ea4ba70c5b4f-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:54.572256 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.572215 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/940b546a-101b-494a-bb85-ea4ba70c5b4f-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:20:54.950344 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.950273 5014 generic.go:356] "Generic (PLEG): container finished" podID="940b546a-101b-494a-bb85-ea4ba70c5b4f" containerID="101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07" exitCode=0 Oct 30 18:20:54.950344 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.950324 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:54.950495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.950359 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"940b546a-101b-494a-bb85-ea4ba70c5b4f","Type":"ContainerDied","Data":"101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07"} Oct 30 18:20:54.950495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.950414 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"940b546a-101b-494a-bb85-ea4ba70c5b4f","Type":"ContainerDied","Data":"4ffe41b4686c1e3e47d0e13e7055dd0fea61ee65199f24adc38b5beae3f38072"} Oct 30 18:20:54.950495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.950438 5014 scope.go:122] "RemoveContainer" containerID="101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07" Oct 30 18:20:54.962861 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.962754 5014 scope.go:122] "RemoveContainer" containerID="101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07" Oct 30 18:20:54.963121 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:20:54.963093 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07\": container with ID starting with 101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07 not found: ID does not exist" containerID="101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07" Oct 30 18:20:54.963340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:54.963301 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07"} err="failed to get container status \"101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07\": rpc error: code = NotFound desc = could not find container \"101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07\": container with ID starting with 101fb974bb29cedb5442fa99b5a9b27e32b661caae420c6c9e021ee51b30ed07 not found: ID does not exist" Oct 30 18:20:55.000540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.000511 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:55.012509 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.012486 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:55.050843 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.050824 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:55.051321 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.051305 5014 cpu_manager.go:414] "containerMap: removing container" podUID="940b546a-101b-494a-bb85-ea4ba70c5b4f" containerName="otc-container" Oct 30 18:20:55.051377 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.051324 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940b546a-101b-494a-bb85-ea4ba70c5b4f" containerName="otc-container" Oct 30 18:20:55.051413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.051406 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="940b546a-101b-494a-bb85-ea4ba70c5b4f" containerName="otc-container" Oct 30 18:20:55.055806 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.055791 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.070201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.070183 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:20:55.070301 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.070216 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:20:55.070301 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.070265 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:20:55.070377 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.070363 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:20:55.070413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.070400 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:20:55.070475 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.070460 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:20:55.075535 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.075514 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:55.075845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.075820 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlskw\" (UniqueName: \"kubernetes.io/projected/e739ea2b-4e08-445d-9cbb-f57a77458e98-kube-api-access-dlskw\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.075972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.075883 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e739ea2b-4e08-445d-9cbb-f57a77458e98-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.076034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.075983 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.076091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.076033 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.076091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.076059 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.076207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.076187 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.141148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.141124 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940b546a-101b-494a-bb85-ea4ba70c5b4f" path="/var/lib/kubelet/pods/940b546a-101b-494a-bb85-ea4ba70c5b4f/volumes" Oct 30 18:20:55.177130 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.177109 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.177240 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.177143 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlskw\" (UniqueName: \"kubernetes.io/projected/e739ea2b-4e08-445d-9cbb-f57a77458e98-kube-api-access-dlskw\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.177363 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.177345 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e739ea2b-4e08-445d-9cbb-f57a77458e98-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.177430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.177398 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.177499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.177452 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.177499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.177478 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.178111 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.178037 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e739ea2b-4e08-445d-9cbb-f57a77458e98-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.179668 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.179651 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.179830 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.179815 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.179879 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.179838 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.180027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.180012 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.199998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.199977 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlskw\" (UniqueName: \"kubernetes.io/projected/e739ea2b-4e08-445d-9cbb-f57a77458e98-kube-api-access-dlskw\") pod \"data-science-collector-collector-0\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.367799 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.367779 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:20:55.666711 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.666633 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:20:55.669799 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:20:55.669773 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode739ea2b_4e08_445d_9cbb_f57a77458e98.slice/crio-61441d08d7b05fbd627806431cea4f97494b02a2e66546beccf52a532bd669a2 WatchSource:0}: Error finding container 61441d08d7b05fbd627806431cea4f97494b02a2e66546beccf52a532bd669a2: Status 404 returned error can't find the container with id 61441d08d7b05fbd627806431cea4f97494b02a2e66546beccf52a532bd669a2 Oct 30 18:20:55.672233 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.672198 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:20:55.672337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.672279 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:20:55.958096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.958026 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e739ea2b-4e08-445d-9cbb-f57a77458e98","Type":"ContainerStarted","Data":"11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8"} Oct 30 18:20:55.958096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:55.958076 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e739ea2b-4e08-445d-9cbb-f57a77458e98","Type":"ContainerStarted","Data":"61441d08d7b05fbd627806431cea4f97494b02a2e66546beccf52a532bd669a2"} Oct 30 18:20:56.000095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:20:56.000047 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.000033983 podStartE2EDuration="1.000033983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:55.999061838 +0000 UTC m=+1468.987402372" watchObservedRunningTime="2026-10-30 18:20:56.000033983 +0000 UTC m=+1468.988374487" Oct 30 18:21:01.214840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.214806 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:21:01.215447 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.215012 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="e739ea2b-4e08-445d-9cbb-f57a77458e98" containerName="otc-container" containerID="cri-o://11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8" gracePeriod=30 Oct 30 18:21:01.471244 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.471189 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:01.529420 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.529393 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e739ea2b-4e08-445d-9cbb-f57a77458e98-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e739ea2b-4e08-445d-9cbb-f57a77458e98-otc-internal\") pod \"e739ea2b-4e08-445d-9cbb-f57a77458e98\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " Oct 30 18:21:01.529585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.529446 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-otlp-tls-certs\") pod \"e739ea2b-4e08-445d-9cbb-f57a77458e98\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " Oct 30 18:21:01.529585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.529497 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-monitoring-tls-certs\") pod \"e739ea2b-4e08-445d-9cbb-f57a77458e98\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " Oct 30 18:21:01.529585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.529527 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-data-science-collector-ta-client-cert\") pod \"e739ea2b-4e08-445d-9cbb-f57a77458e98\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " Oct 30 18:21:01.529585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.529567 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e739ea2b-4e08-445d-9cbb-f57a77458e98-kube-api-access-dlskw\" (UniqueName: \"kubernetes.io/projected/e739ea2b-4e08-445d-9cbb-f57a77458e98-kube-api-access-dlskw\") pod \"e739ea2b-4e08-445d-9cbb-f57a77458e98\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " Oct 30 18:21:01.529850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.529617 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-tls-certs\") pod \"e739ea2b-4e08-445d-9cbb-f57a77458e98\" (UID: \"e739ea2b-4e08-445d-9cbb-f57a77458e98\") " Oct 30 18:21:01.529850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.529786 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e739ea2b-4e08-445d-9cbb-f57a77458e98-otc-internal" pod "e739ea2b-4e08-445d-9cbb-f57a77458e98" (UID: "e739ea2b-4e08-445d-9cbb-f57a77458e98"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:01.529968 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.529903 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e739ea2b-4e08-445d-9cbb-f57a77458e98-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:21:01.531978 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.531949 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e739ea2b-4e08-445d-9cbb-f57a77458e98-kube-api-access-dlskw" pod "e739ea2b-4e08-445d-9cbb-f57a77458e98" (UID: "e739ea2b-4e08-445d-9cbb-f57a77458e98"). InnerVolumeSpecName "kube-api-access-dlskw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:01.532251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.532231 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-data-science-collector-ta-client-cert" pod "e739ea2b-4e08-445d-9cbb-f57a77458e98" (UID: "e739ea2b-4e08-445d-9cbb-f57a77458e98"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:01.532354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.532306 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-monitoring-tls-certs" pod "e739ea2b-4e08-445d-9cbb-f57a77458e98" (UID: "e739ea2b-4e08-445d-9cbb-f57a77458e98"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:01.532424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.532401 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-otlp-tls-certs" pod "e739ea2b-4e08-445d-9cbb-f57a77458e98" (UID: "e739ea2b-4e08-445d-9cbb-f57a77458e98"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:01.532463 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.532444 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-tls-certs" pod "e739ea2b-4e08-445d-9cbb-f57a77458e98" (UID: "e739ea2b-4e08-445d-9cbb-f57a77458e98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:01.631157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.631132 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:21:01.631157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.631154 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:21:01.631297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.631165 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlskw\" (UniqueName: \"kubernetes.io/projected/e739ea2b-4e08-445d-9cbb-f57a77458e98-kube-api-access-dlskw\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:21:01.631297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.631175 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:21:01.631297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.631184 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e739ea2b-4e08-445d-9cbb-f57a77458e98-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:21:01.988418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.988388 5014 generic.go:356] "Generic (PLEG): container finished" podID="e739ea2b-4e08-445d-9cbb-f57a77458e98" containerID="11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8" exitCode=0 Oct 30 18:21:01.988618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.988436 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:01.988618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.988472 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e739ea2b-4e08-445d-9cbb-f57a77458e98","Type":"ContainerDied","Data":"11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8"} Oct 30 18:21:01.988618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.988531 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e739ea2b-4e08-445d-9cbb-f57a77458e98","Type":"ContainerDied","Data":"61441d08d7b05fbd627806431cea4f97494b02a2e66546beccf52a532bd669a2"} Oct 30 18:21:01.988618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.988555 5014 scope.go:122] "RemoveContainer" containerID="11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8" Oct 30 18:21:01.998329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.998315 5014 scope.go:122] "RemoveContainer" containerID="11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8" Oct 30 18:21:01.998624 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:21:01.998579 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8\": container with ID starting with 11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8 not found: ID does not exist" containerID="11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8" Oct 30 18:21:01.998727 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:01.998631 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8"} err="failed to get container status \"11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8\": rpc error: code = NotFound desc = could not find container \"11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8\": container with ID starting with 11572e8a8f3dc2ee0a21160cff74740c1b004e8e3116ac5a91d52e7e463e0ac8 not found: ID does not exist" Oct 30 18:21:02.062136 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.062113 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:21:02.073760 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.073740 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:21:02.116415 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.116397 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:21:02.116796 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.116782 5014 cpu_manager.go:414] "containerMap: removing container" podUID="e739ea2b-4e08-445d-9cbb-f57a77458e98" containerName="otc-container" Oct 30 18:21:02.116796 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.116795 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e739ea2b-4e08-445d-9cbb-f57a77458e98" containerName="otc-container" Oct 30 18:21:02.116909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.116872 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e739ea2b-4e08-445d-9cbb-f57a77458e98" containerName="otc-container" Oct 30 18:21:02.121334 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.121318 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.135921 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.135901 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:21:02.136020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.136011 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:21:02.136074 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.136022 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:21:02.136074 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.136033 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:21:02.136074 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.136053 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:21:02.136074 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.136067 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:21:02.149252 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.149230 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:21:02.235625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.235579 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9hbp\" (UniqueName: \"kubernetes.io/projected/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-kube-api-access-b9hbp\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.235925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.235649 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.235925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.235728 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.235925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.235761 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.235925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.235862 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.235925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.235888 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.337512 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.337276 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.337512 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.337362 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.337512 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.337475 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.337781 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.337524 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.337781 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.337567 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9hbp\" (UniqueName: \"kubernetes.io/projected/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-kube-api-access-b9hbp\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.337781 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.337671 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.341053 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.341026 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.341750 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.341726 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.342632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.342589 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.345520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.345491 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.348508 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.348483 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.360983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.360958 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9hbp\" (UniqueName: \"kubernetes.io/projected/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-kube-api-access-b9hbp\") pod \"data-science-collector-collector-0\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.433612 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.433570 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:21:02.729951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.729929 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:21:02.732022 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:21:02.731994 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ccc783_bfef_4648_bf42_bb13e4ec8e28.slice/crio-830c32ff38c0c61cf7506788f7862ae37ce5c1176f0342549a6070a05f8b84be WatchSource:0}: Error finding container 830c32ff38c0c61cf7506788f7862ae37ce5c1176f0342549a6070a05f8b84be: Status 404 returned error can't find the container with id 830c32ff38c0c61cf7506788f7862ae37ce5c1176f0342549a6070a05f8b84be Oct 30 18:21:02.734080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.734051 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:21:02.734157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.734114 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:21:02.995918 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.995837 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d1ccc783-bfef-4648-bf42-bb13e4ec8e28","Type":"ContainerStarted","Data":"91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9"} Oct 30 18:21:02.995918 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:02.995888 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d1ccc783-bfef-4648-bf42-bb13e4ec8e28","Type":"ContainerStarted","Data":"830c32ff38c0c61cf7506788f7862ae37ce5c1176f0342549a6070a05f8b84be"} Oct 30 18:21:03.040418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:03.040359 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.040345544 podStartE2EDuration="1.040345544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:03.0388133 +0000 UTC m=+1476.027153787" watchObservedRunningTime="2026-10-30 18:21:03.040345544 +0000 UTC m=+1476.028686031" Oct 30 18:21:03.141289 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:03.141254 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e739ea2b-4e08-445d-9cbb-f57a77458e98" path="/var/lib/kubelet/pods/e739ea2b-4e08-445d-9cbb-f57a77458e98/volumes" Oct 30 18:21:13.334710 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:21:13.334670 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:21:27.272792 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:27.272765 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:21:27.275617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:21:27.275567 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:21:43.334022 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:21:43.333986 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:22:13.335895 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:22:13.335855 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:22:43.334642 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:22:43.334587 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:23:13.335712 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:23:13.335665 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:23:38.248859 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.248826 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs"] Oct 30 18:23:38.253079 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.253059 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" Oct 30 18:23:38.268726 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.268703 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"openshift-service-ca.crt\"" Oct 30 18:23:38.268857 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.268743 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-tenant-metrics\"/\"default-dockercfg-zbgh5\"" Oct 30 18:23:38.280093 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.280071 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"kube-root-ca.crt\"" Oct 30 18:23:38.281653 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.281635 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs"] Oct 30 18:23:38.321303 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.321276 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/677e3338-3677-4629-9550-c9fff312dcc6-data\") pod \"tenant-metrics-exporter-5cf544fb97-95hzs\" (UID: \"677e3338-3677-4629-9550-c9fff312dcc6\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" Oct 30 18:23:38.321436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.321319 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w7d5\" (UniqueName: \"kubernetes.io/projected/677e3338-3677-4629-9550-c9fff312dcc6-kube-api-access-8w7d5\") pod \"tenant-metrics-exporter-5cf544fb97-95hzs\" (UID: \"677e3338-3677-4629-9550-c9fff312dcc6\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" Oct 30 18:23:38.422459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.422434 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/677e3338-3677-4629-9550-c9fff312dcc6-data\") pod \"tenant-metrics-exporter-5cf544fb97-95hzs\" (UID: \"677e3338-3677-4629-9550-c9fff312dcc6\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" Oct 30 18:23:38.422561 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.422473 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7d5\" (UniqueName: \"kubernetes.io/projected/677e3338-3677-4629-9550-c9fff312dcc6-kube-api-access-8w7d5\") pod \"tenant-metrics-exporter-5cf544fb97-95hzs\" (UID: \"677e3338-3677-4629-9550-c9fff312dcc6\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" Oct 30 18:23:38.422806 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.422790 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/677e3338-3677-4629-9550-c9fff312dcc6-data\") pod \"tenant-metrics-exporter-5cf544fb97-95hzs\" (UID: \"677e3338-3677-4629-9550-c9fff312dcc6\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" Oct 30 18:23:38.445097 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.445074 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7d5\" (UniqueName: \"kubernetes.io/projected/677e3338-3677-4629-9550-c9fff312dcc6-kube-api-access-8w7d5\") pod \"tenant-metrics-exporter-5cf544fb97-95hzs\" (UID: \"677e3338-3677-4629-9550-c9fff312dcc6\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" Oct 30 18:23:38.566167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.566146 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" Oct 30 18:23:38.862348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.862276 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs"] Oct 30 18:23:38.865473 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:23:38.865446 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677e3338_3677_4629_9550_c9fff312dcc6.slice/crio-ca9511746f7da1603edeb6c47f5c9e30d7c2b45255b67a2a479e909ea1d07dd8 WatchSource:0}: Error finding container ca9511746f7da1603edeb6c47f5c9e30d7c2b45255b67a2a479e909ea1d07dd8: Status 404 returned error can't find the container with id ca9511746f7da1603edeb6c47f5c9e30d7c2b45255b67a2a479e909ea1d07dd8 Oct 30 18:23:38.867319 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:38.867304 5014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:23:39.726524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:39.726482 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" event={"ID":"677e3338-3677-4629-9550-c9fff312dcc6","Type":"ContainerStarted","Data":"ca9511746f7da1603edeb6c47f5c9e30d7c2b45255b67a2a479e909ea1d07dd8"} Oct 30 18:23:40.922098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:40.921246 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:23:40.922098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:40.921497 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d1ccc783-bfef-4648-bf42-bb13e4ec8e28" containerName="otc-container" containerID="cri-o://91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9" gracePeriod=30 Oct 30 18:23:42.743430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:42.743345 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" event={"ID":"677e3338-3677-4629-9550-c9fff312dcc6","Type":"ContainerStarted","Data":"1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0"} Oct 30 18:23:42.781793 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:42.781744 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" podStartSLOduration=1.218106941 podStartE2EDuration="4.781730462s" totalImagesPullingTime="3.563623521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:42.781060517 +0000 UTC m=+1635.769401005" watchObservedRunningTime="2026-10-30 18:23:42.781730462 +0000 UTC m=+1635.770070949" Oct 30 18:23:43.269535 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.269515 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:43.337068 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:23:43.336986 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:23:43.366393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.366368 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-kube-api-access-b9hbp\" (UniqueName: \"kubernetes.io/projected/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-kube-api-access-b9hbp\") pod \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " Oct 30 18:23:43.366559 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.366404 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-monitoring-tls-certs\") pod \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " Oct 30 18:23:43.366559 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.366459 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otc-internal\") pod \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " Oct 30 18:23:43.366559 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.366537 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otlp-tls-certs\") pod \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " Oct 30 18:23:43.366766 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.366586 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-data-science-collector-ta-client-cert\") pod \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " Oct 30 18:23:43.366766 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.366635 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-tls-certs\") pod \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\" (UID: \"d1ccc783-bfef-4648-bf42-bb13e4ec8e28\") " Oct 30 18:23:43.366939 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.366914 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otc-internal" pod "d1ccc783-bfef-4648-bf42-bb13e4ec8e28" (UID: "d1ccc783-bfef-4648-bf42-bb13e4ec8e28"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:43.368667 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.368641 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-kube-api-access-b9hbp" pod "d1ccc783-bfef-4648-bf42-bb13e4ec8e28" (UID: "d1ccc783-bfef-4648-bf42-bb13e4ec8e28"). InnerVolumeSpecName "kube-api-access-b9hbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:43.368847 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.368822 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otlp-tls-certs" pod "d1ccc783-bfef-4648-bf42-bb13e4ec8e28" (UID: "d1ccc783-bfef-4648-bf42-bb13e4ec8e28"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:43.368958 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.368859 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-data-science-collector-ta-client-cert" pod "d1ccc783-bfef-4648-bf42-bb13e4ec8e28" (UID: "d1ccc783-bfef-4648-bf42-bb13e4ec8e28"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:43.369193 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.369178 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-monitoring-tls-certs" pod "d1ccc783-bfef-4648-bf42-bb13e4ec8e28" (UID: "d1ccc783-bfef-4648-bf42-bb13e4ec8e28"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:43.369265 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.369196 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-tls-certs" pod "d1ccc783-bfef-4648-bf42-bb13e4ec8e28" (UID: "d1ccc783-bfef-4648-bf42-bb13e4ec8e28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:43.468299 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.468278 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:23:43.468299 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.468299 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:23:43.468437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.468308 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:23:43.468437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.468329 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9hbp\" (UniqueName: \"kubernetes.io/projected/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-kube-api-access-b9hbp\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:23:43.468437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.468342 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:23:43.468437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.468350 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d1ccc783-bfef-4648-bf42-bb13e4ec8e28-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:23:43.749543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.749481 5014 generic.go:356] "Generic (PLEG): container finished" podID="d1ccc783-bfef-4648-bf42-bb13e4ec8e28" containerID="91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9" exitCode=0 Oct 30 18:23:43.749543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.749536 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:43.750041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.749566 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d1ccc783-bfef-4648-bf42-bb13e4ec8e28","Type":"ContainerDied","Data":"91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9"} Oct 30 18:23:43.750041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.749650 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d1ccc783-bfef-4648-bf42-bb13e4ec8e28","Type":"ContainerDied","Data":"830c32ff38c0c61cf7506788f7862ae37ce5c1176f0342549a6070a05f8b84be"} Oct 30 18:23:43.750041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.749677 5014 scope.go:122] "RemoveContainer" containerID="91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9" Oct 30 18:23:43.758877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.758863 5014 scope.go:122] "RemoveContainer" containerID="91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9" Oct 30 18:23:43.759153 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:23:43.759116 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9\": container with ID starting with 91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9 not found: ID does not exist" containerID="91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9" Oct 30 18:23:43.759229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.759163 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9"} err="failed to get container status \"91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9\": rpc error: code = NotFound desc = could not find container \"91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9\": container with ID starting with 91e749bb0174333f035109410a8b853399610531169c738fd06d1b25c14252a9 not found: ID does not exist" Oct 30 18:23:43.798638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.798586 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:23:43.802329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.802312 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:23:43.841754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.841735 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:23:43.842171 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.842155 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d1ccc783-bfef-4648-bf42-bb13e4ec8e28" containerName="otc-container" Oct 30 18:23:43.842171 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.842171 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ccc783-bfef-4648-bf42-bb13e4ec8e28" containerName="otc-container" Oct 30 18:23:43.842297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.842248 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ccc783-bfef-4648-bf42-bb13e4ec8e28" containerName="otc-container" Oct 30 18:23:43.846486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.846471 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:43.860910 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.860893 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:23:43.861032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.860960 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:23:43.861032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.860971 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:23:43.861032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.860986 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:23:43.861032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.860994 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:23:43.861174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.861035 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:23:43.867722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.867701 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:23:43.973266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.973241 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:43.973372 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.973268 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f2bd4d61-52e9-4271-9d01-01144a045696-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:43.973372 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.973287 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:43.973491 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.973454 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fmbx\" (UniqueName: \"kubernetes.io/projected/f2bd4d61-52e9-4271-9d01-01144a045696-kube-api-access-4fmbx\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:43.973551 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.973535 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:43.973632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:43.973574 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.074353 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.074332 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmbx\" (UniqueName: \"kubernetes.io/projected/f2bd4d61-52e9-4271-9d01-01144a045696-kube-api-access-4fmbx\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.074459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.074374 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.074459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.074397 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.074459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.074430 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.074459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.074452 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f2bd4d61-52e9-4271-9d01-01144a045696-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.074709 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.074469 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.075384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.075361 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f2bd4d61-52e9-4271-9d01-01144a045696-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.077030 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.077003 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.077122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.077066 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.077122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.077109 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.077198 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.077179 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.096692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.096670 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fmbx\" (UniqueName: \"kubernetes.io/projected/f2bd4d61-52e9-4271-9d01-01144a045696-kube-api-access-4fmbx\") pod \"data-science-collector-collector-0\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.158996 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.158978 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:23:44.452857 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.452823 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:23:44.455269 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:23:44.455241 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bd4d61_52e9_4271_9d01_01144a045696.slice/crio-a6eaa057b38bdb9b73cf97e5599c42072a04383b2abac8971105766904d2af14 WatchSource:0}: Error finding container a6eaa057b38bdb9b73cf97e5599c42072a04383b2abac8971105766904d2af14: Status 404 returned error can't find the container with id a6eaa057b38bdb9b73cf97e5599c42072a04383b2abac8971105766904d2af14 Oct 30 18:23:44.457512 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.457482 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:23:44.457642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.457534 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:23:44.756399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.756311 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f2bd4d61-52e9-4271-9d01-01144a045696","Type":"ContainerStarted","Data":"6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24"} Oct 30 18:23:44.756399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.756383 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f2bd4d61-52e9-4271-9d01-01144a045696","Type":"ContainerStarted","Data":"a6eaa057b38bdb9b73cf97e5599c42072a04383b2abac8971105766904d2af14"} Oct 30 18:23:44.798208 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:44.798161 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.7981483 podStartE2EDuration="1.7981483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:44.796804025 +0000 UTC m=+1637.785144513" watchObservedRunningTime="2026-10-30 18:23:44.7981483 +0000 UTC m=+1637.786488788" Oct 30 18:23:45.141443 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:23:45.141408 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ccc783-bfef-4648-bf42-bb13e4ec8e28" path="/var/lib/kubelet/pods/d1ccc783-bfef-4648-bf42-bb13e4ec8e28/volumes" Oct 30 18:24:13.333377 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:24:13.333287 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:24:43.335951 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:24:43.335911 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:24:45.103558 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:45.103525 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs"] Oct 30 18:24:45.104043 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:45.103753 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" podUID="677e3338-3677-4629-9550-c9fff312dcc6" containerName="tenant-metrics-exporter" containerID="cri-o://1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0" gracePeriod=30 Oct 30 18:24:45.358670 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:45.358616 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" Oct 30 18:24:45.439764 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:45.439722 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677e3338-3677-4629-9550-c9fff312dcc6-data\" (UniqueName: \"kubernetes.io/empty-dir/677e3338-3677-4629-9550-c9fff312dcc6-data\") pod \"677e3338-3677-4629-9550-c9fff312dcc6\" (UID: \"677e3338-3677-4629-9550-c9fff312dcc6\") " Oct 30 18:24:45.439885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:45.439808 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677e3338-3677-4629-9550-c9fff312dcc6-kube-api-access-8w7d5\" (UniqueName: \"kubernetes.io/projected/677e3338-3677-4629-9550-c9fff312dcc6-kube-api-access-8w7d5\") pod \"677e3338-3677-4629-9550-c9fff312dcc6\" (UID: \"677e3338-3677-4629-9550-c9fff312dcc6\") " Oct 30 18:24:45.440064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:45.440044 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677e3338-3677-4629-9550-c9fff312dcc6-data" pod "677e3338-3677-4629-9550-c9fff312dcc6" (UID: "677e3338-3677-4629-9550-c9fff312dcc6"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:45.442136 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:45.442117 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677e3338-3677-4629-9550-c9fff312dcc6-kube-api-access-8w7d5" pod "677e3338-3677-4629-9550-c9fff312dcc6" (UID: "677e3338-3677-4629-9550-c9fff312dcc6"). InnerVolumeSpecName "kube-api-access-8w7d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:45.540938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:45.540914 5014 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/677e3338-3677-4629-9550-c9fff312dcc6-data\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:24:45.540938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:45.540937 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w7d5\" (UniqueName: \"kubernetes.io/projected/677e3338-3677-4629-9550-c9fff312dcc6-kube-api-access-8w7d5\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.041881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:46.041847 5014 generic.go:356] "Generic (PLEG): container finished" podID="677e3338-3677-4629-9550-c9fff312dcc6" containerID="1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0" exitCode=0 Oct 30 18:24:46.042027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:46.041886 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" event={"ID":"677e3338-3677-4629-9550-c9fff312dcc6","Type":"ContainerDied","Data":"1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0"} Oct 30 18:24:46.042027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:46.041941 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" event={"ID":"677e3338-3677-4629-9550-c9fff312dcc6","Type":"ContainerDied","Data":"ca9511746f7da1603edeb6c47f5c9e30d7c2b45255b67a2a479e909ea1d07dd8"} Oct 30 18:24:46.042027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:46.041967 5014 scope.go:122] "RemoveContainer" containerID="1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0" Oct 30 18:24:46.042027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:46.041895 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs" Oct 30 18:24:46.051426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:46.051411 5014 scope.go:122] "RemoveContainer" containerID="1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0" Oct 30 18:24:46.051712 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:24:46.051692 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0\": container with ID starting with 1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0 not found: ID does not exist" containerID="1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0" Oct 30 18:24:46.051766 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:46.051721 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0"} err="failed to get container status \"1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0\": rpc error: code = NotFound desc = could not find container \"1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0\": container with ID starting with 1c848fe7517a09d1c275b995bba84ad45ff90ef83fb899d40f1547aabc26dad0 not found: ID does not exist" Oct 30 18:24:46.091396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:46.091370 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs"] Oct 30 18:24:46.095909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:46.095887 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-95hzs"] Oct 30 18:24:47.142120 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:47.142093 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677e3338-3677-4629-9550-c9fff312dcc6" path="/var/lib/kubelet/pods/677e3338-3677-4629-9550-c9fff312dcc6/volumes" Oct 30 18:24:57.610065 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.610034 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter"] Oct 30 18:24:57.610520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.610419 5014 cpu_manager.go:414] "containerMap: removing container" podUID="677e3338-3677-4629-9550-c9fff312dcc6" containerName="tenant-metrics-exporter" Oct 30 18:24:57.610520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.610431 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677e3338-3677-4629-9550-c9fff312dcc6" containerName="tenant-metrics-exporter" Oct 30 18:24:57.610520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.610519 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="677e3338-3677-4629-9550-c9fff312dcc6" containerName="tenant-metrics-exporter" Oct 30 18:24:57.613811 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.613793 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:57.631571 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.630173 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-dcgm-https-18e361e16eb86214\"/\"dcgm-https-exporter\"" Oct 30 18:24:57.632863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.632841 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter"] Oct 30 18:24:57.638541 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.638523 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-dcgm-https-18e361e16eb86214\"/\"dcgm-https-tls\"" Oct 30 18:24:57.641880 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.641848 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-dcgm-https-18e361e16eb86214\"/\"default-dockercfg-7jsjv\"" Oct 30 18:24:57.729227 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.729205 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-web-config\") pod \"dcgm-https-exporter\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:57.729334 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.729254 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-tls\") pod \"dcgm-https-exporter\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:57.729378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.729361 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-data\") pod \"dcgm-https-exporter\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:57.829712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.829689 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-web-config\") pod \"dcgm-https-exporter\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:57.829814 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.829744 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-tls\") pod \"dcgm-https-exporter\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:57.829814 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.829798 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-data\") pod \"dcgm-https-exporter\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:57.830192 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.830172 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-data\") pod \"dcgm-https-exporter\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:57.830413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.830398 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-web-config\") pod \"dcgm-https-exporter\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:57.832239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.832223 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-tls\") pod \"dcgm-https-exporter\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:57.926708 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:57.926660 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:58.225119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:58.225045 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter"] Oct 30 18:24:58.229477 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:24:58.229448 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc583c6a9_61fc_4c0f_b219_7bafe060e3d4.slice/crio-e2d4d06af33a2e135c250f7207e59b9961d6f36828bf7ce44dd806bc047ecca6 WatchSource:0}: Error finding container e2d4d06af33a2e135c250f7207e59b9961d6f36828bf7ce44dd806bc047ecca6: Status 404 returned error can't find the container with id e2d4d06af33a2e135c250f7207e59b9961d6f36828bf7ce44dd806bc047ecca6 Oct 30 18:24:59.105782 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:59.105747 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" event={"ID":"c583c6a9-61fc-4c0f-b219-7bafe060e3d4","Type":"ContainerStarted","Data":"bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6"} Oct 30 18:24:59.106248 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:59.105801 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" event={"ID":"c583c6a9-61fc-4c0f-b219-7bafe060e3d4","Type":"ContainerStarted","Data":"e2d4d06af33a2e135c250f7207e59b9961d6f36828bf7ce44dd806bc047ecca6"} Oct 30 18:24:59.106248 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:59.105836 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:24:59.145323 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:24:59.145277 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" podStartSLOduration=2.145264604 podStartE2EDuration="2.145264604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:59.143955587 +0000 UTC m=+1712.132296074" watchObservedRunningTime="2026-10-30 18:24:59.145264604 +0000 UTC m=+1712.133605091" Oct 30 18:25:05.112935 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:05.112906 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:25:13.333580 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:25:13.333541 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:25:19.548892 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:19.548859 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:25:19.550363 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:19.549678 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="f2bd4d61-52e9-4271-9d01-01144a045696" containerName="otc-container" containerID="cri-o://6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24" gracePeriod=30 Oct 30 18:25:21.902705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:21.902684 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:25:22.018729 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.018661 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-otlp-tls-certs\") pod \"f2bd4d61-52e9-4271-9d01-01144a045696\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " Oct 30 18:25:22.018729 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.018707 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-monitoring-tls-certs\") pod \"f2bd4d61-52e9-4271-9d01-01144a045696\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " Oct 30 18:25:22.018902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.018730 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-data-science-collector-ta-client-cert\") pod \"f2bd4d61-52e9-4271-9d01-01144a045696\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " Oct 30 18:25:22.018902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.018778 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-tls-certs\") pod \"f2bd4d61-52e9-4271-9d01-01144a045696\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " Oct 30 18:25:22.018902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.018874 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2bd4d61-52e9-4271-9d01-01144a045696-kube-api-access-4fmbx\" (UniqueName: \"kubernetes.io/projected/f2bd4d61-52e9-4271-9d01-01144a045696-kube-api-access-4fmbx\") pod \"f2bd4d61-52e9-4271-9d01-01144a045696\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " Oct 30 18:25:22.019063 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.018943 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2bd4d61-52e9-4271-9d01-01144a045696-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f2bd4d61-52e9-4271-9d01-01144a045696-otc-internal\") pod \"f2bd4d61-52e9-4271-9d01-01144a045696\" (UID: \"f2bd4d61-52e9-4271-9d01-01144a045696\") " Oct 30 18:25:22.019515 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.019370 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2bd4d61-52e9-4271-9d01-01144a045696-otc-internal" pod "f2bd4d61-52e9-4271-9d01-01144a045696" (UID: "f2bd4d61-52e9-4271-9d01-01144a045696"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:22.021089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.021053 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bd4d61-52e9-4271-9d01-01144a045696-kube-api-access-4fmbx" pod "f2bd4d61-52e9-4271-9d01-01144a045696" (UID: "f2bd4d61-52e9-4271-9d01-01144a045696"). InnerVolumeSpecName "kube-api-access-4fmbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:22.021089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.021082 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-monitoring-tls-certs" pod "f2bd4d61-52e9-4271-9d01-01144a045696" (UID: "f2bd4d61-52e9-4271-9d01-01144a045696"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:22.021269 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.021106 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-otlp-tls-certs" pod "f2bd4d61-52e9-4271-9d01-01144a045696" (UID: "f2bd4d61-52e9-4271-9d01-01144a045696"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:22.021522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.021500 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-data-science-collector-ta-client-cert" pod "f2bd4d61-52e9-4271-9d01-01144a045696" (UID: "f2bd4d61-52e9-4271-9d01-01144a045696"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:22.021654 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.021529 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-tls-certs" pod "f2bd4d61-52e9-4271-9d01-01144a045696" (UID: "f2bd4d61-52e9-4271-9d01-01144a045696"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:22.120220 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.120199 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:25:22.120220 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.120218 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:25:22.120367 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.120228 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:25:22.120367 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.120237 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2bd4d61-52e9-4271-9d01-01144a045696-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:25:22.120367 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.120246 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fmbx\" (UniqueName: \"kubernetes.io/projected/f2bd4d61-52e9-4271-9d01-01144a045696-kube-api-access-4fmbx\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:25:22.120367 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.120253 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f2bd4d61-52e9-4271-9d01-01144a045696-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:25:22.219383 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.219358 5014 generic.go:356] "Generic (PLEG): container finished" podID="f2bd4d61-52e9-4271-9d01-01144a045696" containerID="6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24" exitCode=0 Oct 30 18:25:22.219479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.219400 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:25:22.219479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.219440 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f2bd4d61-52e9-4271-9d01-01144a045696","Type":"ContainerDied","Data":"6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24"} Oct 30 18:25:22.219555 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.219484 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f2bd4d61-52e9-4271-9d01-01144a045696","Type":"ContainerDied","Data":"a6eaa057b38bdb9b73cf97e5599c42072a04383b2abac8971105766904d2af14"} Oct 30 18:25:22.219555 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.219505 5014 scope.go:122] "RemoveContainer" containerID="6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24" Oct 30 18:25:22.229010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.228995 5014 scope.go:122] "RemoveContainer" containerID="6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24" Oct 30 18:25:22.229249 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:25:22.229230 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24\": container with ID starting with 6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24 not found: ID does not exist" containerID="6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24" Oct 30 18:25:22.229300 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.229257 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24"} err="failed to get container status \"6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24\": rpc error: code = NotFound desc = could not find container \"6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24\": container with ID starting with 6e8ef1a3ed985ff1d4bed51a19c063e2022e409293cbb94eee62cb812995bd24 not found: ID does not exist" Oct 30 18:25:22.269387 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.269337 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:25:22.273311 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:22.273290 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:25:23.142129 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:25:23.142099 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2bd4d61-52e9-4271-9d01-01144a045696" path="/var/lib/kubelet/pods/f2bd4d61-52e9-4271-9d01-01144a045696/volumes" Oct 30 18:25:43.333483 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:25:43.333443 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:26:13.333932 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:26:13.333893 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:26:27.301834 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:26:27.301807 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:26:27.307299 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:26:27.307278 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:26:43.334280 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:26:43.334242 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:27:13.336582 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:27:13.336493 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:27:30.280384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.280353 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:30.280980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.280956 5014 cpu_manager.go:414] "containerMap: removing container" podUID="f2bd4d61-52e9-4271-9d01-01144a045696" containerName="otc-container" Oct 30 18:27:30.280980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.280981 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2bd4d61-52e9-4271-9d01-01144a045696" containerName="otc-container" Oct 30 18:27:30.281155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.281077 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2bd4d61-52e9-4271-9d01-01144a045696" containerName="otc-container" Oct 30 18:27:30.284207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.284185 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.310208 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.310180 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:27:30.310529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.310496 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:27:30.310529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.310508 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:27:30.310529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.310526 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:27:30.310777 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.310540 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:27:30.310777 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.310558 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:27:30.323704 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.323680 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:30.388260 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.388231 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.388361 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.388285 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.388408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.388369 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.388408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.388398 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6c3289a-7570-41c7-87d7-dec22a33bdee-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.388511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.388426 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.388568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.388526 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdcg8\" (UniqueName: \"kubernetes.io/projected/e6c3289a-7570-41c7-87d7-dec22a33bdee-kube-api-access-zdcg8\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.488941 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.488913 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.489056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.488947 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.489056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.488967 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6c3289a-7570-41c7-87d7-dec22a33bdee-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.489056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.488988 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.489223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.489197 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdcg8\" (UniqueName: \"kubernetes.io/projected/e6c3289a-7570-41c7-87d7-dec22a33bdee-kube-api-access-zdcg8\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.489294 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.489270 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.489745 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.489724 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6c3289a-7570-41c7-87d7-dec22a33bdee-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.492066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.492042 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.492168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.492110 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.492168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.492147 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.496893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.496872 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.512251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.512229 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdcg8\" (UniqueName: \"kubernetes.io/projected/e6c3289a-7570-41c7-87d7-dec22a33bdee-kube-api-access-zdcg8\") pod \"data-science-collector-collector-1\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.596884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.596867 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.892810 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.892728 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:30.897509 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:27:30.897483 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c3289a_7570_41c7_87d7_dec22a33bdee.slice/crio-0a62acbaecb67e33b37a0628b915f8fc5f7633aa421fd89f04524b8ef7e3f1ba WatchSource:0}: Error finding container 0a62acbaecb67e33b37a0628b915f8fc5f7633aa421fd89f04524b8ef7e3f1ba: Status 404 returned error can't find the container with id 0a62acbaecb67e33b37a0628b915f8fc5f7633aa421fd89f04524b8ef7e3f1ba Oct 30 18:27:30.899861 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.899830 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:27:30.899944 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:30.899890 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:27:31.843578 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:31.843536 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e6c3289a-7570-41c7-87d7-dec22a33bdee","Type":"ContainerStarted","Data":"bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad"} Oct 30 18:27:31.844060 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:31.843613 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e6c3289a-7570-41c7-87d7-dec22a33bdee","Type":"ContainerStarted","Data":"0a62acbaecb67e33b37a0628b915f8fc5f7633aa421fd89f04524b8ef7e3f1ba"} Oct 30 18:27:31.885566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:31.885520 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.885504805 podStartE2EDuration="1.885504805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:31.884838028 +0000 UTC m=+1864.873178516" watchObservedRunningTime="2026-10-30 18:27:31.885504805 +0000 UTC m=+1864.873845293" Oct 30 18:27:34.029382 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.029340 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter"] Oct 30 18:27:34.029965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.029564 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" podUID="c583c6a9-61fc-4c0f-b219-7bafe060e3d4" containerName="dcgm-https-exporter" containerID="cri-o://bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6" gracePeriod=30 Oct 30 18:27:34.297792 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.297734 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:27:34.318421 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.318399 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-web-config\" (UniqueName: \"kubernetes.io/configmap/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-web-config\") pod \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " Oct 30 18:27:34.318570 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.318429 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-tls\" (UniqueName: \"kubernetes.io/secret/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-tls\") pod \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " Oct 30 18:27:34.318570 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.318454 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-data\" (UniqueName: \"kubernetes.io/empty-dir/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-data\") pod \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\" (UID: \"c583c6a9-61fc-4c0f-b219-7bafe060e3d4\") " Oct 30 18:27:34.318919 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.318899 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-data" pod "c583c6a9-61fc-4c0f-b219-7bafe060e3d4" (UID: "c583c6a9-61fc-4c0f-b219-7bafe060e3d4"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:34.320570 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.320552 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-tls" pod "c583c6a9-61fc-4c0f-b219-7bafe060e3d4" (UID: "c583c6a9-61fc-4c0f-b219-7bafe060e3d4"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:34.323484 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.323462 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-web-config" pod "c583c6a9-61fc-4c0f-b219-7bafe060e3d4" (UID: "c583c6a9-61fc-4c0f-b219-7bafe060e3d4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:34.419469 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.419438 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:27:34.419469 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.419466 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:27:34.419657 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.419476 5014 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c583c6a9-61fc-4c0f-b219-7bafe060e3d4-data\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:27:34.859501 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.859467 5014 generic.go:356] "Generic (PLEG): container finished" podID="c583c6a9-61fc-4c0f-b219-7bafe060e3d4" containerID="bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6" exitCode=0 Oct 30 18:27:34.859695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.859518 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" Oct 30 18:27:34.859695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.859560 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" event={"ID":"c583c6a9-61fc-4c0f-b219-7bafe060e3d4","Type":"ContainerDied","Data":"bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6"} Oct 30 18:27:34.859695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.859646 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter" event={"ID":"c583c6a9-61fc-4c0f-b219-7bafe060e3d4","Type":"ContainerDied","Data":"e2d4d06af33a2e135c250f7207e59b9961d6f36828bf7ce44dd806bc047ecca6"} Oct 30 18:27:34.859695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.859678 5014 scope.go:122] "RemoveContainer" containerID="bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6" Oct 30 18:27:34.869807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.869790 5014 scope.go:122] "RemoveContainer" containerID="bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6" Oct 30 18:27:34.870076 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:27:34.870060 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6\": container with ID starting with bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6 not found: ID does not exist" containerID="bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6" Oct 30 18:27:34.870144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.870084 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6"} err="failed to get container status \"bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6\": rpc error: code = NotFound desc = could not find container \"bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6\": container with ID starting with bdbac907628d368fbc700fe300ef255fb5877125c9cd92e1e9266130a6552af6 not found: ID does not exist" Oct 30 18:27:34.908502 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.908481 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter"] Oct 30 18:27:34.920351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:34.920330 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-dcgm-https-18e361e16eb86214/dcgm-https-exporter"] Oct 30 18:27:35.141469 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:35.141401 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c583c6a9-61fc-4c0f-b219-7bafe060e3d4" path="/var/lib/kubelet/pods/c583c6a9-61fc-4c0f-b219-7bafe060e3d4/volumes" Oct 30 18:27:43.333002 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:27:43.332963 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:27:52.188489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:52.188453 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:52.189290 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:52.188726 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="e6c3289a-7570-41c7-87d7-dec22a33bdee" containerName="otc-container" containerID="cri-o://bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad" gracePeriod=30 Oct 30 18:27:53.821492 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.821455 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq"] Oct 30 18:27:53.822001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.821983 5014 cpu_manager.go:414] "containerMap: removing container" podUID="c583c6a9-61fc-4c0f-b219-7bafe060e3d4" containerName="dcgm-https-exporter" Oct 30 18:27:53.822055 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.822003 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c583c6a9-61fc-4c0f-b219-7bafe060e3d4" containerName="dcgm-https-exporter" Oct 30 18:27:53.822090 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.822081 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="c583c6a9-61fc-4c0f-b219-7bafe060e3d4" containerName="dcgm-https-exporter" Oct 30 18:27:53.825371 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.825353 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" Oct 30 18:27:53.842932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.842900 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"openshift-service-ca.crt\"" Oct 30 18:27:53.844903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.844878 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq"] Oct 30 18:27:53.854571 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.854550 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"kube-root-ca.crt\"" Oct 30 18:27:53.869649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.869618 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-tenant-metrics\"/\"default-dockercfg-2nfmv\"" Oct 30 18:27:53.878105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.878076 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-data\") pod \"tenant-metrics-exporter-5cf544fb97-grgqq\" (UID: \"24c0eb0d-aa65-471e-8f2e-876bf3bde44d\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" Oct 30 18:27:53.878236 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.878121 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm8nx\" (UniqueName: \"kubernetes.io/projected/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-kube-api-access-bm8nx\") pod \"tenant-metrics-exporter-5cf544fb97-grgqq\" (UID: \"24c0eb0d-aa65-471e-8f2e-876bf3bde44d\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" Oct 30 18:27:53.979550 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.979512 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-data\") pod \"tenant-metrics-exporter-5cf544fb97-grgqq\" (UID: \"24c0eb0d-aa65-471e-8f2e-876bf3bde44d\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" Oct 30 18:27:53.979550 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.979557 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8nx\" (UniqueName: \"kubernetes.io/projected/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-kube-api-access-bm8nx\") pod \"tenant-metrics-exporter-5cf544fb97-grgqq\" (UID: \"24c0eb0d-aa65-471e-8f2e-876bf3bde44d\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" Oct 30 18:27:53.979958 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:53.979937 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-data\") pod \"tenant-metrics-exporter-5cf544fb97-grgqq\" (UID: \"24c0eb0d-aa65-471e-8f2e-876bf3bde44d\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" Oct 30 18:27:54.002540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.002511 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8nx\" (UniqueName: \"kubernetes.io/projected/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-kube-api-access-bm8nx\") pod \"tenant-metrics-exporter-5cf544fb97-grgqq\" (UID: \"24c0eb0d-aa65-471e-8f2e-876bf3bde44d\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" Oct 30 18:27:54.140231 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.140128 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" Oct 30 18:27:54.437359 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.437262 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq"] Oct 30 18:27:54.442044 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:27:54.442013 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c0eb0d_aa65_471e_8f2e_876bf3bde44d.slice/crio-74fd4b1d9d97d2ccdcab7b5e9937f7755bdb194824621689a6a68436e5c081af WatchSource:0}: Error finding container 74fd4b1d9d97d2ccdcab7b5e9937f7755bdb194824621689a6a68436e5c081af: Status 404 returned error can't find the container with id 74fd4b1d9d97d2ccdcab7b5e9937f7755bdb194824621689a6a68436e5c081af Oct 30 18:27:54.531122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.531099 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.585953 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-otlp-tls-certs\") pod \"e6c3289a-7570-41c7-87d7-dec22a33bdee\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.586009 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-monitoring-tls-certs\") pod \"e6c3289a-7570-41c7-87d7-dec22a33bdee\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.586065 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c3289a-7570-41c7-87d7-dec22a33bdee-kube-api-access-zdcg8\" (UniqueName: \"kubernetes.io/projected/e6c3289a-7570-41c7-87d7-dec22a33bdee-kube-api-access-zdcg8\") pod \"e6c3289a-7570-41c7-87d7-dec22a33bdee\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.586147 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-tls-certs\") pod \"e6c3289a-7570-41c7-87d7-dec22a33bdee\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.586193 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-data-science-collector-ta-client-cert\") pod \"e6c3289a-7570-41c7-87d7-dec22a33bdee\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.586236 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c3289a-7570-41c7-87d7-dec22a33bdee-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6c3289a-7570-41c7-87d7-dec22a33bdee-otc-internal\") pod \"e6c3289a-7570-41c7-87d7-dec22a33bdee\" (UID: \"e6c3289a-7570-41c7-87d7-dec22a33bdee\") " Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.587492 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c3289a-7570-41c7-87d7-dec22a33bdee-otc-internal" pod "e6c3289a-7570-41c7-87d7-dec22a33bdee" (UID: "e6c3289a-7570-41c7-87d7-dec22a33bdee"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.589251 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-otlp-tls-certs" pod "e6c3289a-7570-41c7-87d7-dec22a33bdee" (UID: "e6c3289a-7570-41c7-87d7-dec22a33bdee"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.590825 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-tls-certs" pod "e6c3289a-7570-41c7-87d7-dec22a33bdee" (UID: "e6c3289a-7570-41c7-87d7-dec22a33bdee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.592967 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-monitoring-tls-certs" pod "e6c3289a-7570-41c7-87d7-dec22a33bdee" (UID: "e6c3289a-7570-41c7-87d7-dec22a33bdee"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.592977 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-data-science-collector-ta-client-cert" pod "e6c3289a-7570-41c7-87d7-dec22a33bdee" (UID: "e6c3289a-7570-41c7-87d7-dec22a33bdee"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:54.593066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.592979 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c3289a-7570-41c7-87d7-dec22a33bdee-kube-api-access-zdcg8" pod "e6c3289a-7570-41c7-87d7-dec22a33bdee" (UID: "e6c3289a-7570-41c7-87d7-dec22a33bdee"). InnerVolumeSpecName "kube-api-access-zdcg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:54.687697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.687588 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:27:54.687697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.687643 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6c3289a-7570-41c7-87d7-dec22a33bdee-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:27:54.687697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.687653 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:27:54.687697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.687663 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:27:54.687697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.687678 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdcg8\" (UniqueName: \"kubernetes.io/projected/e6c3289a-7570-41c7-87d7-dec22a33bdee-kube-api-access-zdcg8\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:27:54.687697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.687692 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c3289a-7570-41c7-87d7-dec22a33bdee-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:27:54.958098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.957996 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" event={"ID":"24c0eb0d-aa65-471e-8f2e-876bf3bde44d","Type":"ContainerStarted","Data":"85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050"} Oct 30 18:27:54.958531 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.958105 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" event={"ID":"24c0eb0d-aa65-471e-8f2e-876bf3bde44d","Type":"ContainerStarted","Data":"74fd4b1d9d97d2ccdcab7b5e9937f7755bdb194824621689a6a68436e5c081af"} Oct 30 18:27:54.959423 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.959397 5014 generic.go:356] "Generic (PLEG): container finished" podID="e6c3289a-7570-41c7-87d7-dec22a33bdee" containerID="bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad" exitCode=0 Oct 30 18:27:54.959526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.959440 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e6c3289a-7570-41c7-87d7-dec22a33bdee","Type":"ContainerDied","Data":"bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad"} Oct 30 18:27:54.959526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.959452 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:54.959526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.959475 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e6c3289a-7570-41c7-87d7-dec22a33bdee","Type":"ContainerDied","Data":"0a62acbaecb67e33b37a0628b915f8fc5f7633aa421fd89f04524b8ef7e3f1ba"} Oct 30 18:27:54.959526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.959496 5014 scope.go:122] "RemoveContainer" containerID="bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad" Oct 30 18:27:54.968659 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.968634 5014 scope.go:122] "RemoveContainer" containerID="bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad" Oct 30 18:27:54.968951 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:27:54.968931 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad\": container with ID starting with bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad not found: ID does not exist" containerID="bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad" Oct 30 18:27:54.969002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.968960 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad"} err="failed to get container status \"bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad\": rpc error: code = NotFound desc = could not find container \"bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad\": container with ID starting with bcfac790a7e83568417829c123cec972ddca7375243ce041b60aa155b65bb8ad not found: ID does not exist" Oct 30 18:27:54.996829 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:54.996782 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" podStartSLOduration=1.9967635110000002 podStartE2EDuration="1.996763511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:54.996188687 +0000 UTC m=+1887.984529177" watchObservedRunningTime="2026-10-30 18:27:54.996763511 +0000 UTC m=+1887.985103997" Oct 30 18:27:55.035620 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.035572 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:55.047401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.047372 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:55.079342 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.079309 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:55.079739 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.079721 5014 cpu_manager.go:414] "containerMap: removing container" podUID="e6c3289a-7570-41c7-87d7-dec22a33bdee" containerName="otc-container" Oct 30 18:27:55.079806 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.079741 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c3289a-7570-41c7-87d7-dec22a33bdee" containerName="otc-container" Oct 30 18:27:55.079879 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.079865 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c3289a-7570-41c7-87d7-dec22a33bdee" containerName="otc-container" Oct 30 18:27:55.084354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.084339 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.090785 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.090753 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrkv\" (UniqueName: \"kubernetes.io/projected/0214e411-a444-4dab-a504-4a7e8eaabdef-kube-api-access-9wrkv\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.090912 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.090810 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.090968 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.090910 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.090968 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.090943 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.091055 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.091003 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.091055 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.091039 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0214e411-a444-4dab-a504-4a7e8eaabdef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.099051 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.099022 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:27:55.099154 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.099126 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:27:55.099212 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.099199 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:27:55.101973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.101956 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:27:55.104619 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.104571 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:55.142452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.142425 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c3289a-7570-41c7-87d7-dec22a33bdee" path="/var/lib/kubelet/pods/e6c3289a-7570-41c7-87d7-dec22a33bdee/volumes" Oct 30 18:27:55.192333 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.192299 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.192446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.192343 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.192446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.192369 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.192568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.192484 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.192568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.192530 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0214e411-a444-4dab-a504-4a7e8eaabdef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.192715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.192585 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrkv\" (UniqueName: \"kubernetes.io/projected/0214e411-a444-4dab-a504-4a7e8eaabdef-kube-api-access-9wrkv\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.193116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.193094 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0214e411-a444-4dab-a504-4a7e8eaabdef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.194900 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.194880 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.195058 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.194881 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.199467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.199446 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:27:55.199467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.199453 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:27:55.204639 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.204620 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.204919 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.204902 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.214879 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.214818 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrkv\" (UniqueName: \"kubernetes.io/projected/0214e411-a444-4dab-a504-4a7e8eaabdef-kube-api-access-9wrkv\") pod \"data-science-collector-collector-1\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.397585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.397551 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:55.696376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.696342 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:55.702758 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:27:55.702725 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0214e411_a444_4dab_a504_4a7e8eaabdef.slice/crio-7598882195f941d6eab24c512185d6f49dddfeca8f68fc8ec75b49a7e7425746 WatchSource:0}: Error finding container 7598882195f941d6eab24c512185d6f49dddfeca8f68fc8ec75b49a7e7425746: Status 404 returned error can't find the container with id 7598882195f941d6eab24c512185d6f49dddfeca8f68fc8ec75b49a7e7425746 Oct 30 18:27:55.704892 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.704865 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:27:55.704976 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.704924 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:27:55.966340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.966256 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0214e411-a444-4dab-a504-4a7e8eaabdef","Type":"ContainerStarted","Data":"095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5"} Oct 30 18:27:55.966340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:55.966312 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0214e411-a444-4dab-a504-4a7e8eaabdef","Type":"ContainerStarted","Data":"7598882195f941d6eab24c512185d6f49dddfeca8f68fc8ec75b49a7e7425746"} Oct 30 18:27:56.008794 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:27:56.008741 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.008726248 podStartE2EDuration="1.008726248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:56.007401839 +0000 UTC m=+1888.995742338" watchObservedRunningTime="2026-10-30 18:27:56.008726248 +0000 UTC m=+1888.997066735" Oct 30 18:28:13.332346 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:28:13.332299 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:28:43.331995 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:28:43.331962 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:29:13.333972 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:29:13.333934 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:29:43.334279 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:29:43.334239 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:30:13.336138 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:30:13.336044 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:30:43.333640 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:30:43.333606 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:31:13.333221 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:31:13.333182 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:31:27.331005 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:31:27.330980 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:31:27.336540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:31:27.336517 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:31:43.332286 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:31:43.332249 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:32:13.334674 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:32:13.334632 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:32:30.714119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:30.714084 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq"] Oct 30 18:32:30.714651 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:30.714293 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" podUID="24c0eb0d-aa65-471e-8f2e-876bf3bde44d" containerName="tenant-metrics-exporter" containerID="cri-o://85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050" gracePeriod=30 Oct 30 18:32:30.813999 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:32:30.813976 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c0eb0d_aa65_471e_8f2e_876bf3bde44d.slice/crio-conmon-85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:32:30.963097 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:30.963074 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" Oct 30 18:32:31.058278 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.058255 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-data\" (UniqueName: \"kubernetes.io/empty-dir/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-data\") pod \"24c0eb0d-aa65-471e-8f2e-876bf3bde44d\" (UID: \"24c0eb0d-aa65-471e-8f2e-876bf3bde44d\") " Oct 30 18:32:31.058382 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.058317 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-kube-api-access-bm8nx\" (UniqueName: \"kubernetes.io/projected/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-kube-api-access-bm8nx\") pod \"24c0eb0d-aa65-471e-8f2e-876bf3bde44d\" (UID: \"24c0eb0d-aa65-471e-8f2e-876bf3bde44d\") " Oct 30 18:32:31.058750 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.058719 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-data" pod "24c0eb0d-aa65-471e-8f2e-876bf3bde44d" (UID: "24c0eb0d-aa65-471e-8f2e-876bf3bde44d"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:31.060651 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.060628 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-kube-api-access-bm8nx" pod "24c0eb0d-aa65-471e-8f2e-876bf3bde44d" (UID: "24c0eb0d-aa65-471e-8f2e-876bf3bde44d"). InnerVolumeSpecName "kube-api-access-bm8nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:31.159242 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.159222 5014 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-data\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:32:31.159327 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.159243 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm8nx\" (UniqueName: \"kubernetes.io/projected/24c0eb0d-aa65-471e-8f2e-876bf3bde44d-kube-api-access-bm8nx\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:32:31.267956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.267925 5014 generic.go:356] "Generic (PLEG): container finished" podID="24c0eb0d-aa65-471e-8f2e-876bf3bde44d" containerID="85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050" exitCode=0 Oct 30 18:32:31.268044 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.267983 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" Oct 30 18:32:31.268044 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.268007 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" event={"ID":"24c0eb0d-aa65-471e-8f2e-876bf3bde44d","Type":"ContainerDied","Data":"85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050"} Oct 30 18:32:31.268122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.268064 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq" event={"ID":"24c0eb0d-aa65-471e-8f2e-876bf3bde44d","Type":"ContainerDied","Data":"74fd4b1d9d97d2ccdcab7b5e9937f7755bdb194824621689a6a68436e5c081af"} Oct 30 18:32:31.268122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.268086 5014 scope.go:122] "RemoveContainer" containerID="85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050" Oct 30 18:32:31.277287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.277271 5014 scope.go:122] "RemoveContainer" containerID="85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050" Oct 30 18:32:31.277559 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:32:31.277539 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050\": container with ID starting with 85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050 not found: ID does not exist" containerID="85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050" Oct 30 18:32:31.277649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.277566 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050"} err="failed to get container status \"85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050\": rpc error: code = NotFound desc = could not find container \"85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050\": container with ID starting with 85dca8ddb5eb01b4a22aa4c91b0beceec22051aa0d4a9fe54062044210742050 not found: ID does not exist" Oct 30 18:32:31.316747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.316702 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq"] Oct 30 18:32:31.320818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:31.320800 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-grgqq"] Oct 30 18:32:33.141751 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:33.141717 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c0eb0d-aa65-471e-8f2e-876bf3bde44d" path="/var/lib/kubelet/pods/24c0eb0d-aa65-471e-8f2e-876bf3bde44d/volumes" Oct 30 18:32:43.334926 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:32:43.334878 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:32:55.869204 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:55.869137 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:32:55.869821 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:55.869408 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0214e411-a444-4dab-a504-4a7e8eaabdef" containerName="otc-container" containerID="cri-o://095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5" gracePeriod=30 Oct 30 18:32:58.238085 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.238062 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.263041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.263018 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0214e411-a444-4dab-a504-4a7e8eaabdef-kube-api-access-9wrkv\" (UniqueName: \"kubernetes.io/projected/0214e411-a444-4dab-a504-4a7e8eaabdef-kube-api-access-9wrkv\") pod \"0214e411-a444-4dab-a504-4a7e8eaabdef\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " Oct 30 18:32:58.263161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.263081 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-data-science-collector-ta-client-cert\") pod \"0214e411-a444-4dab-a504-4a7e8eaabdef\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " Oct 30 18:32:58.263209 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.263169 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-monitoring-tls-certs\") pod \"0214e411-a444-4dab-a504-4a7e8eaabdef\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " Oct 30 18:32:58.263253 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.263233 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-tls-certs\") pod \"0214e411-a444-4dab-a504-4a7e8eaabdef\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " Oct 30 18:32:58.263289 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.263255 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-otlp-tls-certs\") pod \"0214e411-a444-4dab-a504-4a7e8eaabdef\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " Oct 30 18:32:58.263341 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.263287 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0214e411-a444-4dab-a504-4a7e8eaabdef-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0214e411-a444-4dab-a504-4a7e8eaabdef-otc-internal\") pod \"0214e411-a444-4dab-a504-4a7e8eaabdef\" (UID: \"0214e411-a444-4dab-a504-4a7e8eaabdef\") " Oct 30 18:32:58.263773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.263747 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0214e411-a444-4dab-a504-4a7e8eaabdef-otc-internal" pod "0214e411-a444-4dab-a504-4a7e8eaabdef" (UID: "0214e411-a444-4dab-a504-4a7e8eaabdef"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:58.265393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.265370 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-otlp-tls-certs" pod "0214e411-a444-4dab-a504-4a7e8eaabdef" (UID: "0214e411-a444-4dab-a504-4a7e8eaabdef"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:58.265820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.265795 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-data-science-collector-ta-client-cert" pod "0214e411-a444-4dab-a504-4a7e8eaabdef" (UID: "0214e411-a444-4dab-a504-4a7e8eaabdef"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:58.265820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.265813 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0214e411-a444-4dab-a504-4a7e8eaabdef-kube-api-access-9wrkv" pod "0214e411-a444-4dab-a504-4a7e8eaabdef" (UID: "0214e411-a444-4dab-a504-4a7e8eaabdef"). InnerVolumeSpecName "kube-api-access-9wrkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:58.265966 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.265894 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-tls-certs" pod "0214e411-a444-4dab-a504-4a7e8eaabdef" (UID: "0214e411-a444-4dab-a504-4a7e8eaabdef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:58.265966 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.265922 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-monitoring-tls-certs" pod "0214e411-a444-4dab-a504-4a7e8eaabdef" (UID: "0214e411-a444-4dab-a504-4a7e8eaabdef"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:58.364010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.363933 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0214e411-a444-4dab-a504-4a7e8eaabdef-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:32:58.364010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.363961 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wrkv\" (UniqueName: \"kubernetes.io/projected/0214e411-a444-4dab-a504-4a7e8eaabdef-kube-api-access-9wrkv\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:32:58.364010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.363974 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:32:58.364010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.363983 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:32:58.364010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.363992 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:32:58.364010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.364003 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214e411-a444-4dab-a504-4a7e8eaabdef-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:32:58.394700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.394670 5014 generic.go:356] "Generic (PLEG): container finished" podID="0214e411-a444-4dab-a504-4a7e8eaabdef" containerID="095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5" exitCode=0 Oct 30 18:32:58.394862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.394715 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.394862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.394756 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0214e411-a444-4dab-a504-4a7e8eaabdef","Type":"ContainerDied","Data":"095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5"} Oct 30 18:32:58.394862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.394820 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0214e411-a444-4dab-a504-4a7e8eaabdef","Type":"ContainerDied","Data":"7598882195f941d6eab24c512185d6f49dddfeca8f68fc8ec75b49a7e7425746"} Oct 30 18:32:58.394862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.394844 5014 scope.go:122] "RemoveContainer" containerID="095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5" Oct 30 18:32:58.404922 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.404905 5014 scope.go:122] "RemoveContainer" containerID="095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5" Oct 30 18:32:58.405174 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:32:58.405157 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5\": container with ID starting with 095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5 not found: ID does not exist" containerID="095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5" Oct 30 18:32:58.405240 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.405191 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5"} err="failed to get container status \"095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5\": rpc error: code = NotFound desc = could not find container \"095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5\": container with ID starting with 095277cf657c7991408a45dd0321b8eb4d599ee83fa24e215363e1ac744e72d5 not found: ID does not exist" Oct 30 18:32:58.446749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.446716 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:32:58.452487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.452454 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:32:58.489409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.489383 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:32:58.489841 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.489823 5014 cpu_manager.go:414] "containerMap: removing container" podUID="0214e411-a444-4dab-a504-4a7e8eaabdef" containerName="otc-container" Oct 30 18:32:58.489841 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.489842 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0214e411-a444-4dab-a504-4a7e8eaabdef" containerName="otc-container" Oct 30 18:32:58.489973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.489871 5014 cpu_manager.go:414] "containerMap: removing container" podUID="24c0eb0d-aa65-471e-8f2e-876bf3bde44d" containerName="tenant-metrics-exporter" Oct 30 18:32:58.489973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.489880 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c0eb0d-aa65-471e-8f2e-876bf3bde44d" containerName="tenant-metrics-exporter" Oct 30 18:32:58.489973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.489944 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0214e411-a444-4dab-a504-4a7e8eaabdef" containerName="otc-container" Oct 30 18:32:58.489973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.489955 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c0eb0d-aa65-471e-8f2e-876bf3bde44d" containerName="tenant-metrics-exporter" Oct 30 18:32:58.494547 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.494530 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.508885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.508861 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:32:58.508885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.508862 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:32:58.509032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.508892 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:32:58.511513 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.511489 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:32:58.514210 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.514182 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:32:58.538853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.538819 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:32:58.565377 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.565351 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.565526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.565385 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srvxg\" (UniqueName: \"kubernetes.io/projected/17072dbd-ebdb-45f0-b28d-6327c75f096c-kube-api-access-srvxg\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.565526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.565419 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.565526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.565467 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.565526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.565507 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.565687 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.565584 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/17072dbd-ebdb-45f0-b28d-6327c75f096c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.609945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.609920 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:32:58.666264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.666203 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/17072dbd-ebdb-45f0-b28d-6327c75f096c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.666264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.666245 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.666386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.666270 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srvxg\" (UniqueName: \"kubernetes.io/projected/17072dbd-ebdb-45f0-b28d-6327c75f096c-kube-api-access-srvxg\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.666386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.666316 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.666386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.666340 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.666386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.666368 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.667131 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.667038 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/17072dbd-ebdb-45f0-b28d-6327c75f096c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.669064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.669041 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.669147 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.669067 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.669147 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.669136 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.669215 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.669170 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.689692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.689667 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srvxg\" (UniqueName: \"kubernetes.io/projected/17072dbd-ebdb-45f0-b28d-6327c75f096c-kube-api-access-srvxg\") pod \"data-science-collector-collector-1\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:58.807568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:58.807536 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:32:59.106416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:59.106366 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:32:59.110720 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:32:59.110689 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17072dbd_ebdb_45f0_b28d_6327c75f096c.slice/crio-e9766fd1937789bbf12399fb39ce898f27da40b21214e59b177c66ef107fefa2 WatchSource:0}: Error finding container e9766fd1937789bbf12399fb39ce898f27da40b21214e59b177c66ef107fefa2: Status 404 returned error can't find the container with id e9766fd1937789bbf12399fb39ce898f27da40b21214e59b177c66ef107fefa2 Oct 30 18:32:59.113054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:59.113022 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:32:59.113145 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:59.113089 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:32:59.142936 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:59.142909 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0214e411-a444-4dab-a504-4a7e8eaabdef" path="/var/lib/kubelet/pods/0214e411-a444-4dab-a504-4a7e8eaabdef/volumes" Oct 30 18:32:59.402871 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:59.402787 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"17072dbd-ebdb-45f0-b28d-6327c75f096c","Type":"ContainerStarted","Data":"0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02"} Oct 30 18:32:59.402871 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:59.402839 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"17072dbd-ebdb-45f0-b28d-6327c75f096c","Type":"ContainerStarted","Data":"e9766fd1937789bbf12399fb39ce898f27da40b21214e59b177c66ef107fefa2"} Oct 30 18:32:59.446505 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:32:59.446444 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.446431035 podStartE2EDuration="1.446431035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:59.444587082 +0000 UTC m=+2192.432927563" watchObservedRunningTime="2026-10-30 18:32:59.446431035 +0000 UTC m=+2192.434771627" Oct 30 18:33:13.335704 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:33:13.335614 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:33:30.906390 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:30.906350 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d"] Oct 30 18:33:30.911948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:30.911924 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d" Oct 30 18:33:30.922844 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:30.922823 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d" Oct 30 18:33:30.927960 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:30.927939 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d"] Oct 30 18:33:31.223075 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:31.223001 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d"] Oct 30 18:33:31.226445 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:33:31.226414 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5aa4e887_60c2_414e_84eb_795cb662fe90.slice/crio-5d82b43eb302900df303aa2fe7c4d8649718a9a82e52f7fd3b128fdb78cb8c38 WatchSource:0}: Error finding container 5d82b43eb302900df303aa2fe7c4d8649718a9a82e52f7fd3b128fdb78cb8c38: Status 404 returned error can't find the container with id 5d82b43eb302900df303aa2fe7c4d8649718a9a82e52f7fd3b128fdb78cb8c38 Oct 30 18:33:31.554834 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:31.554798 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d" event={"ID":"5aa4e887-60c2-414e-84eb-795cb662fe90","Type":"ContainerStarted","Data":"e975c4a7157d9f201a4bc78f8eb67f1acf5ee4349c70a95338ead0769ee52dc7"} Oct 30 18:33:31.554996 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:31.554845 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d" event={"ID":"5aa4e887-60c2-414e-84eb-795cb662fe90","Type":"ContainerStarted","Data":"5d82b43eb302900df303aa2fe7c4d8649718a9a82e52f7fd3b128fdb78cb8c38"} Oct 30 18:33:31.592787 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:31.592739 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d" podStartSLOduration=1.592726301 podStartE2EDuration="1.592726301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:31.59183711 +0000 UTC m=+2224.580177598" watchObservedRunningTime="2026-10-30 18:33:31.592726301 +0000 UTC m=+2224.581066788" Oct 30 18:33:41.607553 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:41.607473 5014 generic.go:356] "Generic (PLEG): container finished" podID="5aa4e887-60c2-414e-84eb-795cb662fe90" containerID="e975c4a7157d9f201a4bc78f8eb67f1acf5ee4349c70a95338ead0769ee52dc7" exitCode=0 Oct 30 18:33:41.607927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:41.607548 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d" event={"ID":"5aa4e887-60c2-414e-84eb-795cb662fe90","Type":"ContainerDied","Data":"e975c4a7157d9f201a4bc78f8eb67f1acf5ee4349c70a95338ead0769ee52dc7"} Oct 30 18:33:42.754151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:42.754126 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d" Oct 30 18:33:43.282017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:43.281983 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0"] Oct 30 18:33:43.282394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:43.282378 5014 cpu_manager.go:414] "containerMap: removing container" podUID="5aa4e887-60c2-414e-84eb-795cb662fe90" containerName="curl" Oct 30 18:33:43.282461 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:43.282395 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa4e887-60c2-414e-84eb-795cb662fe90" containerName="curl" Oct 30 18:33:43.282497 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:43.282486 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa4e887-60c2-414e-84eb-795cb662fe90" containerName="curl" Oct 30 18:33:43.285585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:43.285568 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0" Oct 30 18:33:43.296134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:43.296115 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0" Oct 30 18:33:43.305548 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:43.304462 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0"] Oct 30 18:33:43.340737 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:33:43.340693 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:33:43.609276 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:43.609247 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0"] Oct 30 18:33:43.613837 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:33:43.613810 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36f9c4cd_4504_4ae4_8465_ba3ee6b90771.slice/crio-9730d0342cde8fa4387f3543e25beb6e4c896d03d884bace854526be03e5f918 WatchSource:0}: Error finding container 9730d0342cde8fa4387f3543e25beb6e4c896d03d884bace854526be03e5f918: Status 404 returned error can't find the container with id 9730d0342cde8fa4387f3543e25beb6e4c896d03d884bace854526be03e5f918 Oct 30 18:33:43.620676 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:43.620659 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d" Oct 30 18:33:43.620765 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:43.620669 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d" event={"ID":"5aa4e887-60c2-414e-84eb-795cb662fe90","Type":"ContainerDied","Data":"5d82b43eb302900df303aa2fe7c4d8649718a9a82e52f7fd3b128fdb78cb8c38"} Oct 30 18:33:43.620765 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:43.620702 5014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d82b43eb302900df303aa2fe7c4d8649718a9a82e52f7fd3b128fdb78cb8c38" Oct 30 18:33:44.633134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:44.633097 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0" event={"ID":"36f9c4cd-4504-4ae4-8465-ba3ee6b90771","Type":"ContainerStarted","Data":"6aad179135995841a24ead2b69c7f2911646243629dade3550ca2737b910caf7"} Oct 30 18:33:44.633134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:44.633144 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0" event={"ID":"36f9c4cd-4504-4ae4-8465-ba3ee6b90771","Type":"ContainerStarted","Data":"9730d0342cde8fa4387f3543e25beb6e4c896d03d884bace854526be03e5f918"} Oct 30 18:33:44.672081 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:44.672037 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0" podStartSLOduration=1.6720246250000002 podStartE2EDuration="1.672024625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:44.670191411 +0000 UTC m=+2237.658531899" watchObservedRunningTime="2026-10-30 18:33:44.672024625 +0000 UTC m=+2237.660365112" Oct 30 18:33:51.665050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:51.665016 5014 generic.go:356] "Generic (PLEG): container finished" podID="36f9c4cd-4504-4ae4-8465-ba3ee6b90771" containerID="6aad179135995841a24ead2b69c7f2911646243629dade3550ca2737b910caf7" exitCode=0 Oct 30 18:33:51.665418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:51.665089 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0" event={"ID":"36f9c4cd-4504-4ae4-8465-ba3ee6b90771","Type":"ContainerDied","Data":"6aad179135995841a24ead2b69c7f2911646243629dade3550ca2737b910caf7"} Oct 30 18:33:52.808554 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:52.808526 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0" Oct 30 18:33:53.677583 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:53.677555 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0" Oct 30 18:33:53.677771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:53.677602 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0" event={"ID":"36f9c4cd-4504-4ae4-8465-ba3ee6b90771","Type":"ContainerDied","Data":"9730d0342cde8fa4387f3543e25beb6e4c896d03d884bace854526be03e5f918"} Oct 30 18:33:53.677771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:33:53.677638 5014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9730d0342cde8fa4387f3543e25beb6e4c896d03d884bace854526be03e5f918" Oct 30 18:34:13.333303 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:34:13.333264 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:34:43.333029 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:34:43.332993 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:35:13.333906 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:35:13.333868 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:35:43.335060 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:35:43.335008 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:36:13.334015 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:36:13.333920 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:36:27.360263 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:36:27.360143 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:36:27.366651 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:36:27.366632 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:36:43.335818 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:36:43.335775 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:37:13.334533 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:37:13.334494 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:37:20.105720 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:20.105680 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0"] Oct 30 18:37:20.108175 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:20.108150 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e3625bcdc0ddc0"] Oct 30 18:37:20.145056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:20.145030 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d"] Oct 30 18:37:20.147151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:20.147120 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e36258f4367e7d"] Oct 30 18:37:21.141331 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:21.141296 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f9c4cd-4504-4ae4-8465-ba3ee6b90771" path="/var/lib/kubelet/pods/36f9c4cd-4504-4ae4-8465-ba3ee6b90771/volumes" Oct 30 18:37:21.141769 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:21.141537 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa4e887-60c2-414e-84eb-795cb662fe90" path="/var/lib/kubelet/pods/5aa4e887-60c2-414e-84eb-795cb662fe90/volumes" Oct 30 18:37:43.334496 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:37:43.334461 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:37:51.711167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:51.711134 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw"] Oct 30 18:37:51.711978 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:51.711479 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" podUID="946d98c4-7d0f-4f45-9f86-b695c9cd5b59" containerName="kube-rbac-proxy" containerID="cri-o://19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3" gracePeriod=30 Oct 30 18:37:51.968107 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:51.968054 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:37:52.047959 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.047928 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58"] Oct 30 18:37:52.048350 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.048286 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" podUID="03e12818-2b17-4a08-a5ce-7acafced4ed2" containerName="thanos-querier" containerID="cri-o://af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5" gracePeriod=30 Oct 30 18:37:52.088414 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.088379 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-client-cert\") pod \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " Oct 30 18:37:52.088549 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.088450 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-api-access-zvcx2\" (UniqueName: \"kubernetes.io/projected/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-api-access-zvcx2\") pod \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " Oct 30 18:37:52.088549 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.088496 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-ca\") pod \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " Oct 30 18:37:52.088549 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.088535 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-tls\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-tls\") pod \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " Oct 30 18:37:52.088723 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.088573 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-rbac-proxy-config\") pod \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\" (UID: \"946d98c4-7d0f-4f45-9f86-b695c9cd5b59\") " Oct 30 18:37:52.089683 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.089658 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-ca" pod "946d98c4-7d0f-4f45-9f86-b695c9cd5b59" (UID: "946d98c4-7d0f-4f45-9f86-b695c9cd5b59"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.091934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.091909 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-rbac-proxy-config" pod "946d98c4-7d0f-4f45-9f86-b695c9cd5b59" (UID: "946d98c4-7d0f-4f45-9f86-b695c9cd5b59"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.093413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.093387 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-client-cert" pod "946d98c4-7d0f-4f45-9f86-b695c9cd5b59" (UID: "946d98c4-7d0f-4f45-9f86-b695c9cd5b59"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.093413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.093397 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-api-access-zvcx2" pod "946d98c4-7d0f-4f45-9f86-b695c9cd5b59" (UID: "946d98c4-7d0f-4f45-9f86-b695c9cd5b59"). InnerVolumeSpecName "kube-api-access-zvcx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:52.093583 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.093569 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-tls" pod "946d98c4-7d0f-4f45-9f86-b695c9cd5b59" (UID: "946d98c4-7d0f-4f45-9f86-b695c9cd5b59"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.191195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.191168 5014 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-rbac-proxy-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.191195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.191195 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.191195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.191205 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvcx2\" (UniqueName: \"kubernetes.io/projected/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-kube-api-access-zvcx2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.191432 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.191217 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-prometheus-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.191432 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.191226 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/946d98c4-7d0f-4f45-9f86-b695c9cd5b59-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.206387 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.202516 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:37:52.206387 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.202828 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="alertmanager" containerID="cri-o://4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed" gracePeriod=120 Oct 30 18:37:52.206387 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.203199 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="config-reloader" containerID="cri-o://c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509" gracePeriod=120 Oct 30 18:37:52.210072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.210045 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:37:52.210392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.210349 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="prometheus" containerID="cri-o://7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb" gracePeriod=600 Oct 30 18:37:52.210392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.210363 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="config-reloader" containerID="cri-o://d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7" gracePeriod=600 Oct 30 18:37:52.210532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.210381 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="thanos-sidecar" containerID="cri-o://42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff" gracePeriod=600 Oct 30 18:37:52.334108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.334081 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" Oct 30 18:37:52.472706 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.472650 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:37:52.494329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.494307 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e12818-2b17-4a08-a5ce-7acafced4ed2-kube-api-access-b9q26\" (UniqueName: \"kubernetes.io/projected/03e12818-2b17-4a08-a5ce-7acafced4ed2-kube-api-access-b9q26\") pod \"03e12818-2b17-4a08-a5ce-7acafced4ed2\" (UID: \"03e12818-2b17-4a08-a5ce-7acafced4ed2\") " Oct 30 18:37:52.496389 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.496371 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e12818-2b17-4a08-a5ce-7acafced4ed2-kube-api-access-b9q26" pod "03e12818-2b17-4a08-a5ce-7acafced4ed2" (UID: "03e12818-2b17-4a08-a5ce-7acafced4ed2"). InnerVolumeSpecName "kube-api-access-b9q26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:52.594992 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.594965 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595001 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595024 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595048 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-secret-prometheus-web-tls-ca\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595071 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-kube-api-access-rnlnl\" (UniqueName: \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-kube-api-access-rnlnl\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595098 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595131 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595157 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-tls-assets\" (UniqueName: \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-tls-assets\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595191 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-thanos-prometheus-http-client-file\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595271 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595310 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config-out\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595407 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.595626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595456 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\" (UID: \"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c\") " Oct 30 18:37:52.596063 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.595849 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9q26\" (UniqueName: \"kubernetes.io/projected/03e12818-2b17-4a08-a5ce-7acafced4ed2-kube-api-access-b9q26\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.596124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.596093 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-1" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.596318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.596291 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-0" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.596318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.596303 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-2" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.598385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.598345 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.598498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.598401 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.598498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.598412 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-secret-prometheus-web-tls-ca" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.599638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.599588 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-thanos-prometheus-http-client-file" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.599760 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.599666 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config-out" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:52.600126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.600104 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-tls-assets" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:52.600317 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.600294 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-kube-api-access-rnlnl" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "kube-api-access-rnlnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:52.600406 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.600314 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.603135 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.603112 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:37:52.614695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.614672 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config" pod "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" (UID: "29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.696239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696211 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.696239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696238 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.696458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696274 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:37:52.696458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696290 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.696458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696303 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.696458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696315 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.696458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696328 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.696458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696348 5014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.696458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696362 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnlnl\" (UniqueName: \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-kube-api-access-rnlnl\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.696458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696377 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.696458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696393 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.696458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696406 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.696458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.696419 5014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.707796 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.707779 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:37:52.797421 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.797393 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-3ece98a829a66f7cef6e6b4403002493a48d56f442e1673d82fff8fda237a009\"" Oct 30 18:37:52.808532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.808506 5014 generic.go:356] "Generic (PLEG): container finished" podID="946d98c4-7d0f-4f45-9f86-b695c9cd5b59" containerID="19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3" exitCode=0 Oct 30 18:37:52.808665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.808570 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" Oct 30 18:37:52.808665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.808628 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" event={"ID":"946d98c4-7d0f-4f45-9f86-b695c9cd5b59","Type":"ContainerDied","Data":"19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3"} Oct 30 18:37:52.808752 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.808689 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw" event={"ID":"946d98c4-7d0f-4f45-9f86-b695c9cd5b59","Type":"ContainerDied","Data":"cef51c7356d0f26417661d80a2d22cdbcc7f857f0e3d27dd1c3eb05d5235592b"} Oct 30 18:37:52.808752 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.808723 5014 scope.go:122] "RemoveContainer" containerID="19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3" Oct 30 18:37:52.812015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.811124 5014 generic.go:356] "Generic (PLEG): container finished" podID="03e12818-2b17-4a08-a5ce-7acafced4ed2" containerID="af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5" exitCode=0 Oct 30 18:37:52.812015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.811219 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" event={"ID":"03e12818-2b17-4a08-a5ce-7acafced4ed2","Type":"ContainerDied","Data":"af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5"} Oct 30 18:37:52.812015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.811250 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" event={"ID":"03e12818-2b17-4a08-a5ce-7acafced4ed2","Type":"ContainerDied","Data":"83bcf91da07152f54e0f0343bfca6aaf3da3d8438ff6c1f7a27b73af3eb36ece"} Oct 30 18:37:52.812015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.811318 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58" Oct 30 18:37:52.819126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.819100 5014 generic.go:356] "Generic (PLEG): container finished" podID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerID="42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff" exitCode=0 Oct 30 18:37:52.819237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.819125 5014 generic.go:356] "Generic (PLEG): container finished" podID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerID="d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7" exitCode=0 Oct 30 18:37:52.819237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.819165 5014 generic.go:356] "Generic (PLEG): container finished" podID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerID="7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb" exitCode=0 Oct 30 18:37:52.819354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.819244 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c","Type":"ContainerDied","Data":"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff"} Oct 30 18:37:52.819354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.819305 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c","Type":"ContainerDied","Data":"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7"} Oct 30 18:37:52.819354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.819347 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c","Type":"ContainerDied","Data":"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb"} Oct 30 18:37:52.819582 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.819383 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c","Type":"ContainerDied","Data":"61e1c42efa1471383a00ec5f5ad00a10cbcb47e1ac0bf93fc514f7d869923b5c"} Oct 30 18:37:52.819582 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.819507 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:37:52.823037 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.822265 5014 scope.go:122] "RemoveContainer" containerID="19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3" Oct 30 18:37:52.827881 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:37:52.825577 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3\": container with ID starting with 19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3 not found: ID does not exist" containerID="19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3" Oct 30 18:37:52.827881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.825638 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3"} err="failed to get container status \"19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3\": rpc error: code = NotFound desc = could not find container \"19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3\": container with ID starting with 19a3bb83d9fbd8e9cfa2466eced8b7c12ec042636c624373f3db4032de157ef3 not found: ID does not exist" Oct 30 18:37:52.827881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.825663 5014 scope.go:122] "RemoveContainer" containerID="af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5" Oct 30 18:37:52.830201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.830084 5014 generic.go:356] "Generic (PLEG): container finished" podID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerID="c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509" exitCode=0 Oct 30 18:37:52.830201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.830136 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"06188f5d-ad56-49f7-baba-8e8b7e1154dc","Type":"ContainerDied","Data":"c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509"} Oct 30 18:37:52.842950 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.842929 5014 scope.go:122] "RemoveContainer" containerID="af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5" Oct 30 18:37:52.843317 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:37:52.843291 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5\": container with ID starting with af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5 not found: ID does not exist" containerID="af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5" Oct 30 18:37:52.843921 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.843455 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5"} err="failed to get container status \"af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5\": rpc error: code = NotFound desc = could not find container \"af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5\": container with ID starting with af41ec91973ec0639755e8105803ed466c6b0f5c472fad6c99850bbe7c92c4a5 not found: ID does not exist" Oct 30 18:37:52.843921 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.843485 5014 scope.go:122] "RemoveContainer" containerID="42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff" Oct 30 18:37:52.852424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.852400 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:37:52.852668 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.852643 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="17072dbd-ebdb-45f0-b28d-6327c75f096c" containerName="otc-container" containerID="cri-o://0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02" gracePeriod=30 Oct 30 18:37:52.854658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.854637 5014 scope.go:122] "RemoveContainer" containerID="d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7" Oct 30 18:37:52.864565 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.864544 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw"] Oct 30 18:37:52.871017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.870951 5014 scope.go:122] "RemoveContainer" containerID="7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb" Oct 30 18:37:52.882877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.882162 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-x55hw"] Oct 30 18:37:52.942441 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.942381 5014 scope.go:122] "RemoveContainer" containerID="a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d" Oct 30 18:37:52.948835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.948811 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58"] Oct 30 18:37:52.960217 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.959837 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5vj58"] Oct 30 18:37:52.967745 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.967728 5014 scope.go:122] "RemoveContainer" containerID="42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff" Oct 30 18:37:52.968142 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:37:52.968105 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff\": container with ID starting with 42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff not found: ID does not exist" containerID="42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff" Oct 30 18:37:52.968268 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.968154 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff"} err="failed to get container status \"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff\": rpc error: code = NotFound desc = could not find container \"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff\": container with ID starting with 42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff not found: ID does not exist" Oct 30 18:37:52.968268 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.968177 5014 scope.go:122] "RemoveContainer" containerID="d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7" Oct 30 18:37:52.968546 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:37:52.968486 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7\": container with ID starting with d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7 not found: ID does not exist" containerID="d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7" Oct 30 18:37:52.968546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.968516 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7"} err="failed to get container status \"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7\": rpc error: code = NotFound desc = could not find container \"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7\": container with ID starting with d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7 not found: ID does not exist" Oct 30 18:37:52.968546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.968530 5014 scope.go:122] "RemoveContainer" containerID="7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb" Oct 30 18:37:52.968824 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:37:52.968807 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb\": container with ID starting with 7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb not found: ID does not exist" containerID="7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb" Oct 30 18:37:52.968867 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.968831 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb"} err="failed to get container status \"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb\": rpc error: code = NotFound desc = could not find container \"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb\": container with ID starting with 7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb not found: ID does not exist" Oct 30 18:37:52.968867 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.968845 5014 scope.go:122] "RemoveContainer" containerID="a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d" Oct 30 18:37:52.969099 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:37:52.969048 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d\": container with ID starting with a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d not found: ID does not exist" containerID="a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d" Oct 30 18:37:52.969099 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.969069 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d"} err="failed to get container status \"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d\": rpc error: code = NotFound desc = could not find container \"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d\": container with ID starting with a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d not found: ID does not exist" Oct 30 18:37:52.969196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.969098 5014 scope.go:122] "RemoveContainer" containerID="42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff" Oct 30 18:37:52.969374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.969339 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff"} err="failed to get container status \"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff\": rpc error: code = NotFound desc = could not find container \"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff\": container with ID starting with 42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff not found: ID does not exist" Oct 30 18:37:52.969374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.969365 5014 scope.go:122] "RemoveContainer" containerID="d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7" Oct 30 18:37:52.969998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.969754 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7"} err="failed to get container status \"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7\": rpc error: code = NotFound desc = could not find container \"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7\": container with ID starting with d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7 not found: ID does not exist" Oct 30 18:37:52.969998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.969780 5014 scope.go:122] "RemoveContainer" containerID="7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb" Oct 30 18:37:52.970130 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.970045 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb"} err="failed to get container status \"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb\": rpc error: code = NotFound desc = could not find container \"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb\": container with ID starting with 7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb not found: ID does not exist" Oct 30 18:37:52.970130 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.970066 5014 scope.go:122] "RemoveContainer" containerID="a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d" Oct 30 18:37:52.970393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.970369 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d"} err="failed to get container status \"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d\": rpc error: code = NotFound desc = could not find container \"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d\": container with ID starting with a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d not found: ID does not exist" Oct 30 18:37:52.970477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.970394 5014 scope.go:122] "RemoveContainer" containerID="42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff" Oct 30 18:37:52.970860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.970826 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff"} err="failed to get container status \"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff\": rpc error: code = NotFound desc = could not find container \"42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff\": container with ID starting with 42893fc315d0e7fc469f4685815f269de96d8b8c6743950abe97f7ec738a1fff not found: ID does not exist" Oct 30 18:37:52.970860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.970855 5014 scope.go:122] "RemoveContainer" containerID="d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7" Oct 30 18:37:52.971157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.971136 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7"} err="failed to get container status \"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7\": rpc error: code = NotFound desc = could not find container \"d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7\": container with ID starting with d7ff96781e05a819793cb64293ae4990239f956f03fb49fe1ab841e53d7ff6c7 not found: ID does not exist" Oct 30 18:37:52.971239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.971188 5014 scope.go:122] "RemoveContainer" containerID="7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb" Oct 30 18:37:52.971498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.971473 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb"} err="failed to get container status \"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb\": rpc error: code = NotFound desc = could not find container \"7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb\": container with ID starting with 7352bcd226f1b2c2e78e5bbad695bce6b8cddf698e57fe092c45371d485fb7cb not found: ID does not exist" Oct 30 18:37:52.971566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.971501 5014 scope.go:122] "RemoveContainer" containerID="a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d" Oct 30 18:37:52.971883 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.971859 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d"} err="failed to get container status \"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d\": rpc error: code = NotFound desc = could not find container \"a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d\": container with ID starting with a1fdddb4ee924a3c60dc5e77d63d173a151aa3c530b7668af766f55e3fcfe59d not found: ID does not exist" Oct 30 18:37:52.994119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:52.994049 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:37:53.005377 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.005350 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:37:53.107996 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.107972 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:53.141378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.141343 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e12818-2b17-4a08-a5ce-7acafced4ed2" path="/var/lib/kubelet/pods/03e12818-2b17-4a08-a5ce-7acafced4ed2/volumes" Oct 30 18:37:53.141899 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.141877 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" path="/var/lib/kubelet/pods/29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c/volumes" Oct 30 18:37:53.142407 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.142388 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946d98c4-7d0f-4f45-9f86-b695c9cd5b59" path="/var/lib/kubelet/pods/946d98c4-7d0f-4f45-9f86-b695c9cd5b59/volumes" Oct 30 18:37:53.201808 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.201783 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-otlp-tls-certs\") pod \"17072dbd-ebdb-45f0-b28d-6327c75f096c\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " Oct 30 18:37:53.201962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.201835 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-tls-certs\") pod \"17072dbd-ebdb-45f0-b28d-6327c75f096c\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " Oct 30 18:37:53.201962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.201863 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-data-science-collector-ta-client-cert\") pod \"17072dbd-ebdb-45f0-b28d-6327c75f096c\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " Oct 30 18:37:53.201962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.201895 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-monitoring-tls-certs\") pod \"17072dbd-ebdb-45f0-b28d-6327c75f096c\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " Oct 30 18:37:53.201962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.201939 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17072dbd-ebdb-45f0-b28d-6327c75f096c-kube-api-access-srvxg\" (UniqueName: \"kubernetes.io/projected/17072dbd-ebdb-45f0-b28d-6327c75f096c-kube-api-access-srvxg\") pod \"17072dbd-ebdb-45f0-b28d-6327c75f096c\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " Oct 30 18:37:53.202179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.201963 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17072dbd-ebdb-45f0-b28d-6327c75f096c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/17072dbd-ebdb-45f0-b28d-6327c75f096c-otc-internal\") pod \"17072dbd-ebdb-45f0-b28d-6327c75f096c\" (UID: \"17072dbd-ebdb-45f0-b28d-6327c75f096c\") " Oct 30 18:37:53.202571 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.202529 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17072dbd-ebdb-45f0-b28d-6327c75f096c-otc-internal" pod "17072dbd-ebdb-45f0-b28d-6327c75f096c" (UID: "17072dbd-ebdb-45f0-b28d-6327c75f096c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:53.204405 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.204375 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17072dbd-ebdb-45f0-b28d-6327c75f096c-kube-api-access-srvxg" pod "17072dbd-ebdb-45f0-b28d-6327c75f096c" (UID: "17072dbd-ebdb-45f0-b28d-6327c75f096c"). InnerVolumeSpecName "kube-api-access-srvxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:53.204405 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.204389 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-tls-certs" pod "17072dbd-ebdb-45f0-b28d-6327c75f096c" (UID: "17072dbd-ebdb-45f0-b28d-6327c75f096c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.204622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.204580 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-otlp-tls-certs" pod "17072dbd-ebdb-45f0-b28d-6327c75f096c" (UID: "17072dbd-ebdb-45f0-b28d-6327c75f096c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.204684 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.204646 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-monitoring-tls-certs" pod "17072dbd-ebdb-45f0-b28d-6327c75f096c" (UID: "17072dbd-ebdb-45f0-b28d-6327c75f096c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.204735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.204718 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-data-science-collector-ta-client-cert" pod "17072dbd-ebdb-45f0-b28d-6327c75f096c" (UID: "17072dbd-ebdb-45f0-b28d-6327c75f096c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.302908 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.302886 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.302908 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.302909 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.303051 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.302918 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.303051 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.302928 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/17072dbd-ebdb-45f0-b28d-6327c75f096c-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.303051 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.302937 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srvxg\" (UniqueName: \"kubernetes.io/projected/17072dbd-ebdb-45f0-b28d-6327c75f096c-kube-api-access-srvxg\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.303051 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.302948 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/17072dbd-ebdb-45f0-b28d-6327c75f096c-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.407150 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.407120 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.54:9093/-/ready\": dial tcp 10.132.0.54:9093: connect: connection refused" Oct 30 18:37:53.836121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.836091 5014 generic.go:356] "Generic (PLEG): container finished" podID="17072dbd-ebdb-45f0-b28d-6327c75f096c" containerID="0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02" exitCode=0 Oct 30 18:37:53.836633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.836134 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:53.836633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.836178 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"17072dbd-ebdb-45f0-b28d-6327c75f096c","Type":"ContainerDied","Data":"0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02"} Oct 30 18:37:53.836633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.836240 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"17072dbd-ebdb-45f0-b28d-6327c75f096c","Type":"ContainerDied","Data":"e9766fd1937789bbf12399fb39ce898f27da40b21214e59b177c66ef107fefa2"} Oct 30 18:37:53.836633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.836259 5014 scope.go:122] "RemoveContainer" containerID="0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02" Oct 30 18:37:53.849014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.848990 5014 scope.go:122] "RemoveContainer" containerID="0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02" Oct 30 18:37:53.849285 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:37:53.849268 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02\": container with ID starting with 0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02 not found: ID does not exist" containerID="0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02" Oct 30 18:37:53.849374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.849294 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02"} err="failed to get container status \"0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02\": rpc error: code = NotFound desc = could not find container \"0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02\": container with ID starting with 0736ffaa068b324ec0a57630e64ab018823aa13157b11abd19c0362a1178dd02 not found: ID does not exist" Oct 30 18:37:53.886255 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.886233 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:37:53.892739 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:53.892719 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:37:55.140539 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:55.140507 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17072dbd-ebdb-45f0-b28d-6327c75f096c" path="/var/lib/kubelet/pods/17072dbd-ebdb-45f0-b28d-6327c75f096c/volumes" Oct 30 18:37:56.610404 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.610345 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:37:56.610980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.610957 5014 cpu_manager.go:414] "containerMap: removing container" podUID="17072dbd-ebdb-45f0-b28d-6327c75f096c" containerName="otc-container" Oct 30 18:37:56.611064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.610982 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17072dbd-ebdb-45f0-b28d-6327c75f096c" containerName="otc-container" Oct 30 18:37:56.611064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.610997 5014 cpu_manager.go:414] "containerMap: removing container" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="init-config-reloader" Oct 30 18:37:56.611064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611002 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="init-config-reloader" Oct 30 18:37:56.611064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611019 5014 cpu_manager.go:414] "containerMap: removing container" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="prometheus" Oct 30 18:37:56.611064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611028 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="prometheus" Oct 30 18:37:56.611064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611037 5014 cpu_manager.go:414] "containerMap: removing container" podUID="03e12818-2b17-4a08-a5ce-7acafced4ed2" containerName="thanos-querier" Oct 30 18:37:56.611064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611042 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e12818-2b17-4a08-a5ce-7acafced4ed2" containerName="thanos-querier" Oct 30 18:37:56.611064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611051 5014 cpu_manager.go:414] "containerMap: removing container" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="config-reloader" Oct 30 18:37:56.611064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611055 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="config-reloader" Oct 30 18:37:56.611064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611064 5014 cpu_manager.go:414] "containerMap: removing container" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="thanos-sidecar" Oct 30 18:37:56.611064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611069 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="thanos-sidecar" Oct 30 18:37:56.611690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611081 5014 cpu_manager.go:414] "containerMap: removing container" podUID="36f9c4cd-4504-4ae4-8465-ba3ee6b90771" containerName="curl" Oct 30 18:37:56.611690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611089 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f9c4cd-4504-4ae4-8465-ba3ee6b90771" containerName="curl" Oct 30 18:37:56.611690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611104 5014 cpu_manager.go:414] "containerMap: removing container" podUID="946d98c4-7d0f-4f45-9f86-b695c9cd5b59" containerName="kube-rbac-proxy" Oct 30 18:37:56.611690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611109 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946d98c4-7d0f-4f45-9f86-b695c9cd5b59" containerName="kube-rbac-proxy" Oct 30 18:37:56.611690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611185 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="17072dbd-ebdb-45f0-b28d-6327c75f096c" containerName="otc-container" Oct 30 18:37:56.611690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611196 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="prometheus" Oct 30 18:37:56.611690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611203 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e12818-2b17-4a08-a5ce-7acafced4ed2" containerName="thanos-querier" Oct 30 18:37:56.611690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611210 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="config-reloader" Oct 30 18:37:56.611690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611217 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="29cf0339-a1d5-4b2c-abeb-01dd1d27bb3c" containerName="thanos-sidecar" Oct 30 18:37:56.611690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611223 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f9c4cd-4504-4ae4-8465-ba3ee6b90771" containerName="curl" Oct 30 18:37:56.611690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.611230 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="946d98c4-7d0f-4f45-9f86-b695c9cd5b59" containerName="kube-rbac-proxy" Oct 30 18:37:56.616530 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.616508 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.636947 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.636915 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:37:56.674454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.674428 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:37:56.674454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.674451 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8ps4g\"" Oct 30 18:37:56.674690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.674567 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:37:56.701516 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.701497 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 18:37:56.724698 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.724674 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f0d2c8db-da77-4026-8eee-205b5350bfb5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.724869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.724847 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.724927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.724884 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbnl\" (UniqueName: \"kubernetes.io/projected/f0d2c8db-da77-4026-8eee-205b5350bfb5-kube-api-access-wwbnl\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.724927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.724913 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.826012 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.825981 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f0d2c8db-da77-4026-8eee-205b5350bfb5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.826151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.826062 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.826151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.826080 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbnl\" (UniqueName: \"kubernetes.io/projected/f0d2c8db-da77-4026-8eee-205b5350bfb5-kube-api-access-wwbnl\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.826151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.826109 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.826839 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.826756 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f0d2c8db-da77-4026-8eee-205b5350bfb5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.828666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.828637 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.828759 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.828700 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.848724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.848701 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbnl\" (UniqueName: \"kubernetes.io/projected/f0d2c8db-da77-4026-8eee-205b5350bfb5-kube-api-access-wwbnl\") pod \"data-science-collector-collector-1\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:56.929873 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:56.929820 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:37:57.228099 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:57.228007 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:37:57.231700 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:37:57.231672 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d2c8db_da77_4026_8eee_205b5350bfb5.slice/crio-b57f54bd44d2bdbf0351f43645032f308a4aba16f9b4f1fbb43997385709f632 WatchSource:0}: Error finding container b57f54bd44d2bdbf0351f43645032f308a4aba16f9b4f1fbb43997385709f632: Status 404 returned error can't find the container with id b57f54bd44d2bdbf0351f43645032f308a4aba16f9b4f1fbb43997385709f632 Oct 30 18:37:57.233761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:57.233733 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:37:57.233859 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:57.233787 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:37:57.862482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:57.862454 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f0d2c8db-da77-4026-8eee-205b5350bfb5","Type":"ContainerStarted","Data":"8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad"} Oct 30 18:37:57.862880 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:57.862507 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f0d2c8db-da77-4026-8eee-205b5350bfb5","Type":"ContainerStarted","Data":"b57f54bd44d2bdbf0351f43645032f308a4aba16f9b4f1fbb43997385709f632"} Oct 30 18:37:57.903936 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:57.903893 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.903879522 podStartE2EDuration="1.903879522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:57.902322017 +0000 UTC m=+2490.890662507" watchObservedRunningTime="2026-10-30 18:37:57.903879522 +0000 UTC m=+2490.892220008" Oct 30 18:37:58.406680 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:37:58.406643 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.54:9093/-/ready\": dial tcp 10.132.0.54:9093: connect: connection refused" Oct 30 18:38:02.460482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.460455 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:38:02.573157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.573127 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-tls-assets\" (UniqueName: \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-tls-assets\") pod \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " Oct 30 18:38:02.573392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.573177 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-kube-api-access-kbrk6\" (UniqueName: \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-kube-api-access-kbrk6\") pod \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " Oct 30 18:38:02.573392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.573212 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-alertmanager-data-science-monitoringstack-db\") pod \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " Oct 30 18:38:02.573392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.573242 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-volume\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-volume\") pod \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " Oct 30 18:38:02.573392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.573259 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-out\" (UniqueName: \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-out\") pod \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " Oct 30 18:38:02.573392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.573282 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-cluster-tls-config\") pod \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " Oct 30 18:38:02.573392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.573348 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-web-config\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-web-config\") pod \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\" (UID: \"06188f5d-ad56-49f7-baba-8e8b7e1154dc\") " Oct 30 18:38:02.573749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.573664 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-alertmanager-data-science-monitoringstack-db" pod "06188f5d-ad56-49f7-baba-8e8b7e1154dc" (UID: "06188f5d-ad56-49f7-baba-8e8b7e1154dc"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:02.575719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.575692 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-tls-assets" pod "06188f5d-ad56-49f7-baba-8e8b7e1154dc" (UID: "06188f5d-ad56-49f7-baba-8e8b7e1154dc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:02.575719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.575703 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-kube-api-access-kbrk6" pod "06188f5d-ad56-49f7-baba-8e8b7e1154dc" (UID: "06188f5d-ad56-49f7-baba-8e8b7e1154dc"). InnerVolumeSpecName "kube-api-access-kbrk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:02.576014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.575995 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-volume" pod "06188f5d-ad56-49f7-baba-8e8b7e1154dc" (UID: "06188f5d-ad56-49f7-baba-8e8b7e1154dc"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:02.576139 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.576115 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-out" pod "06188f5d-ad56-49f7-baba-8e8b7e1154dc" (UID: "06188f5d-ad56-49f7-baba-8e8b7e1154dc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:02.580131 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.580106 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-cluster-tls-config" pod "06188f5d-ad56-49f7-baba-8e8b7e1154dc" (UID: "06188f5d-ad56-49f7-baba-8e8b7e1154dc"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:02.587531 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.587508 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-web-config" pod "06188f5d-ad56-49f7-baba-8e8b7e1154dc" (UID: "06188f5d-ad56-49f7-baba-8e8b7e1154dc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:02.674915 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.674893 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:38:02.674992 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.674916 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:38:02.674992 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.674931 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbrk6\" (UniqueName: \"kubernetes.io/projected/06188f5d-ad56-49f7-baba-8e8b7e1154dc-kube-api-access-kbrk6\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:38:02.674992 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.674941 5014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:38:02.674992 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.674952 5014 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-volume\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:38:02.674992 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.674960 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06188f5d-ad56-49f7-baba-8e8b7e1154dc-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:38:02.674992 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.674967 5014 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06188f5d-ad56-49f7-baba-8e8b7e1154dc-cluster-tls-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:38:02.889682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.889623 5014 generic.go:356] "Generic (PLEG): container finished" podID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerID="4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed" exitCode=0 Oct 30 18:38:02.889770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.889675 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"06188f5d-ad56-49f7-baba-8e8b7e1154dc","Type":"ContainerDied","Data":"4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed"} Oct 30 18:38:02.889770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.889707 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:38:02.889770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.889755 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"06188f5d-ad56-49f7-baba-8e8b7e1154dc","Type":"ContainerDied","Data":"31142d658a081f81548aa0484b1627471a8f47b15201e297d990be0760ea8cf9"} Oct 30 18:38:02.889872 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.889794 5014 scope.go:122] "RemoveContainer" containerID="c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509" Oct 30 18:38:02.899003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.898986 5014 scope.go:122] "RemoveContainer" containerID="4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed" Oct 30 18:38:02.907396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.907380 5014 scope.go:122] "RemoveContainer" containerID="68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930" Oct 30 18:38:02.914775 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.914759 5014 scope.go:122] "RemoveContainer" containerID="c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509" Oct 30 18:38:02.915046 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:38:02.915019 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509\": container with ID starting with c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509 not found: ID does not exist" containerID="c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509" Oct 30 18:38:02.915089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.915055 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509"} err="failed to get container status \"c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509\": rpc error: code = NotFound desc = could not find container \"c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509\": container with ID starting with c58e4af0794915ec024fc61f0524e34fcae8444306ddf48185c578d91021a509 not found: ID does not exist" Oct 30 18:38:02.915089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.915072 5014 scope.go:122] "RemoveContainer" containerID="4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed" Oct 30 18:38:02.915348 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:38:02.915333 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed\": container with ID starting with 4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed not found: ID does not exist" containerID="4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed" Oct 30 18:38:02.915415 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.915352 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed"} err="failed to get container status \"4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed\": rpc error: code = NotFound desc = could not find container \"4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed\": container with ID starting with 4b21fccd89f47a591aa1d3cdc59cd69113c28aa35626fb268ba5e4570d07a8ed not found: ID does not exist" Oct 30 18:38:02.915415 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.915365 5014 scope.go:122] "RemoveContainer" containerID="68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930" Oct 30 18:38:02.915633 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:38:02.915617 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930\": container with ID starting with 68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930 not found: ID does not exist" containerID="68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930" Oct 30 18:38:02.915672 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.915639 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930"} err="failed to get container status \"68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930\": rpc error: code = NotFound desc = could not find container \"68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930\": container with ID starting with 68e49689b676f3f40e0837f6e6d3c437d5644095e71787019e98103f4086c930 not found: ID does not exist" Oct 30 18:38:02.941043 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.941023 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:38:02.946064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:02.946045 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:38:03.141085 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:03.141027 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" path="/var/lib/kubelet/pods/06188f5d-ad56-49f7-baba-8e8b7e1154dc/volumes" Oct 30 18:38:13.332136 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:38:13.332100 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:38:43.332092 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:38:43.332052 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:38:52.341224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.341193 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:38:52.341747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.341426 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="f0d2c8db-da77-4026-8eee-205b5350bfb5" containerName="otc-container" containerID="cri-o://8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad" gracePeriod=30 Oct 30 18:38:52.589523 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.589498 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:52.634888 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.634823 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d2c8db-da77-4026-8eee-205b5350bfb5-kube-api-access-wwbnl\" (UniqueName: \"kubernetes.io/projected/f0d2c8db-da77-4026-8eee-205b5350bfb5-kube-api-access-wwbnl\") pod \"f0d2c8db-da77-4026-8eee-205b5350bfb5\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " Oct 30 18:38:52.634888 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.634871 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-monitoring-tls-certs\") pod \"f0d2c8db-da77-4026-8eee-205b5350bfb5\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " Oct 30 18:38:52.635298 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.635280 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d2c8db-da77-4026-8eee-205b5350bfb5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f0d2c8db-da77-4026-8eee-205b5350bfb5-otc-internal\") pod \"f0d2c8db-da77-4026-8eee-205b5350bfb5\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " Oct 30 18:38:52.635399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.635311 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-otlp-tls-certs\") pod \"f0d2c8db-da77-4026-8eee-205b5350bfb5\" (UID: \"f0d2c8db-da77-4026-8eee-205b5350bfb5\") " Oct 30 18:38:52.635897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.635871 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d2c8db-da77-4026-8eee-205b5350bfb5-otc-internal" pod "f0d2c8db-da77-4026-8eee-205b5350bfb5" (UID: "f0d2c8db-da77-4026-8eee-205b5350bfb5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:52.637687 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.637588 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-monitoring-tls-certs" pod "f0d2c8db-da77-4026-8eee-205b5350bfb5" (UID: "f0d2c8db-da77-4026-8eee-205b5350bfb5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:52.637824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.637792 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d2c8db-da77-4026-8eee-205b5350bfb5-kube-api-access-wwbnl" pod "f0d2c8db-da77-4026-8eee-205b5350bfb5" (UID: "f0d2c8db-da77-4026-8eee-205b5350bfb5"). InnerVolumeSpecName "kube-api-access-wwbnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:52.638216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.638194 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-otlp-tls-certs" pod "f0d2c8db-da77-4026-8eee-205b5350bfb5" (UID: "f0d2c8db-da77-4026-8eee-205b5350bfb5"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:52.736855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.736829 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f0d2c8db-da77-4026-8eee-205b5350bfb5-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.736855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.736853 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.737003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.736863 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwbnl\" (UniqueName: \"kubernetes.io/projected/f0d2c8db-da77-4026-8eee-205b5350bfb5-kube-api-access-wwbnl\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.737003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:52.736873 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d2c8db-da77-4026-8eee-205b5350bfb5-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:38:53.127756 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:53.127718 5014 generic.go:356] "Generic (PLEG): container finished" podID="f0d2c8db-da77-4026-8eee-205b5350bfb5" containerID="8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad" exitCode=0 Oct 30 18:38:53.127932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:53.127777 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:53.127932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:53.127780 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f0d2c8db-da77-4026-8eee-205b5350bfb5","Type":"ContainerDied","Data":"8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad"} Oct 30 18:38:53.127932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:53.127837 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f0d2c8db-da77-4026-8eee-205b5350bfb5","Type":"ContainerDied","Data":"b57f54bd44d2bdbf0351f43645032f308a4aba16f9b4f1fbb43997385709f632"} Oct 30 18:38:53.127932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:53.127869 5014 scope.go:122] "RemoveContainer" containerID="8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad" Oct 30 18:38:53.138266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:53.138241 5014 scope.go:122] "RemoveContainer" containerID="8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad" Oct 30 18:38:53.138553 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:38:53.138529 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad\": container with ID starting with 8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad not found: ID does not exist" containerID="8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad" Oct 30 18:38:53.138668 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:53.138564 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad"} err="failed to get container status \"8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad\": rpc error: code = NotFound desc = could not find container \"8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad\": container with ID starting with 8d1e69fff02b5dfda0cf5fe1b29818b2d91f9ae3d367c8c1c96c08c8b0e8acad not found: ID does not exist" Oct 30 18:38:53.184838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:53.184812 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:38:53.198300 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:53.198276 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:38:55.141071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:55.141031 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d2c8db-da77-4026-8eee-205b5350bfb5" path="/var/lib/kubelet/pods/f0d2c8db-da77-4026-8eee-205b5350bfb5/volumes" Oct 30 18:38:56.337741 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.337711 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:38:56.338124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.338015 5014 cpu_manager.go:414] "containerMap: removing container" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="init-config-reloader" Oct 30 18:38:56.338124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.338029 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="init-config-reloader" Oct 30 18:38:56.338124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.338048 5014 cpu_manager.go:414] "containerMap: removing container" podUID="f0d2c8db-da77-4026-8eee-205b5350bfb5" containerName="otc-container" Oct 30 18:38:56.338124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.338056 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d2c8db-da77-4026-8eee-205b5350bfb5" containerName="otc-container" Oct 30 18:38:56.338124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.338073 5014 cpu_manager.go:414] "containerMap: removing container" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="alertmanager" Oct 30 18:38:56.338124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.338082 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="alertmanager" Oct 30 18:38:56.338124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.338108 5014 cpu_manager.go:414] "containerMap: removing container" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="config-reloader" Oct 30 18:38:56.338124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.338115 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="config-reloader" Oct 30 18:38:56.338453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.338219 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d2c8db-da77-4026-8eee-205b5350bfb5" containerName="otc-container" Oct 30 18:38:56.338453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.338231 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="alertmanager" Oct 30 18:38:56.338453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.338239 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="06188f5d-ad56-49f7-baba-8e8b7e1154dc" containerName="config-reloader" Oct 30 18:38:56.342934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.342908 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.359229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.359131 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 18:38:56.370808 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.370780 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:38:56.371012 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.370994 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:38:56.371209 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.371187 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:38:56.373440 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.373408 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:38:56.373557 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.373541 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-96qc2\"" Oct 30 18:38:56.373635 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.373539 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:38:56.462575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.462542 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe1297f2-5abb-4173-a2c0-c47950da58ed-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.462731 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.462582 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jmhc\" (UniqueName: \"kubernetes.io/projected/fe1297f2-5abb-4173-a2c0-c47950da58ed-kube-api-access-4jmhc\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.462800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.462722 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.462853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.462791 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.564098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.564070 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.564230 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.564135 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe1297f2-5abb-4173-a2c0-c47950da58ed-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.564230 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.564159 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jmhc\" (UniqueName: \"kubernetes.io/projected/fe1297f2-5abb-4173-a2c0-c47950da58ed-kube-api-access-4jmhc\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.564230 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.564207 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.564942 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.564871 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe1297f2-5abb-4173-a2c0-c47950da58ed-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.566620 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.566584 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.566718 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.566645 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.587054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.587034 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jmhc\" (UniqueName: \"kubernetes.io/projected/fe1297f2-5abb-4173-a2c0-c47950da58ed-kube-api-access-4jmhc\") pod \"data-science-collector-collector-0\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.660519 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.660456 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:56.958274 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.958203 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:38:56.963230 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:38:56.963204 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1297f2_5abb_4173_a2c0_c47950da58ed.slice/crio-2b8dfb4732c25cfe369f7d56c741649e4685b8489ad45c30270b846779e5a7bd WatchSource:0}: Error finding container 2b8dfb4732c25cfe369f7d56c741649e4685b8489ad45c30270b846779e5a7bd: Status 404 returned error can't find the container with id 2b8dfb4732c25cfe369f7d56c741649e4685b8489ad45c30270b846779e5a7bd Oct 30 18:38:56.965289 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.965259 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:38:56.965369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:56.965323 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:38:57.040802 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.040758 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:38:57.044933 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.044913 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.062851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.062827 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:38:57.062851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.062827 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:38:57.063288 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.063270 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:38:57.072826 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.072803 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:38:57.072826 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.072810 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:38:57.073008 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.072848 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:38:57.073008 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.072896 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:38:57.073111 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.073012 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-c2ffr\"" Oct 30 18:38:57.074606 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.074567 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:38:57.150553 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.150519 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fe1297f2-5abb-4173-a2c0-c47950da58ed","Type":"ContainerStarted","Data":"2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d"} Oct 30 18:38:57.150749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.150567 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fe1297f2-5abb-4173-a2c0-c47950da58ed","Type":"ContainerStarted","Data":"2b8dfb4732c25cfe369f7d56c741649e4685b8489ad45c30270b846779e5a7bd"} Oct 30 18:38:57.168694 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.168665 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.168825 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.168717 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.168896 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.168822 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.168896 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.168851 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.169004 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.168911 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.169004 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.168929 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.169004 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.168949 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkd7n\" (UniqueName: \"kubernetes.io/projected/764fef65-8431-4f94-9658-c637818790ec-kube-api-access-mkd7n\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.169167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.169015 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.190676 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.190634 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.190622967 podStartE2EDuration="1.190622967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:57.19052982 +0000 UTC m=+2550.178870319" watchObservedRunningTime="2026-10-30 18:38:57.190622967 +0000 UTC m=+2550.178963448" Oct 30 18:38:57.269665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.269576 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.269665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.269644 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.269878 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.269680 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.269878 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.269701 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.269878 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.269720 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.269878 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.269768 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.269878 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.269783 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.269878 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.269802 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkd7n\" (UniqueName: \"kubernetes.io/projected/764fef65-8431-4f94-9658-c637818790ec-kube-api-access-mkd7n\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.270640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.270614 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.270909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.270880 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.271085 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.271066 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.271234 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.271211 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.272180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.272160 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.272295 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.272277 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.272345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.272282 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:57.292382 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:38:57.292363 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkd7n\" (UniqueName: \"kubernetes.io/projected/764fef65-8431-4f94-9658-c637818790ec-kube-api-access-mkd7n\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:05.327311 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:05.327271 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:05.355886 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:05.355855 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:05.428177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:05.428147 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:05.428277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:05.428259 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:05.442486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:05.442462 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:05.541883 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:05.541858 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:05.568221 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:05.568202 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:05.764030 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:05.763979 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:06.093115 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:06.093089 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:39:06.192516 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:06.192486 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"764fef65-8431-4f94-9658-c637818790ec","Type":"ContainerStarted","Data":"3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93"} Oct 30 18:39:06.192652 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:06.192577 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"764fef65-8431-4f94-9658-c637818790ec","Type":"ContainerStarted","Data":"8efc5ed41adfe8535dbc0c616d48620a86bb8424aa74fffb279aea3dd5056efe"} Oct 30 18:39:07.199376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:07.199282 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"764fef65-8431-4f94-9658-c637818790ec","Type":"ContainerStarted","Data":"3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f"} Oct 30 18:39:07.199376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:07.199339 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"764fef65-8431-4f94-9658-c637818790ec","Type":"ContainerStarted","Data":"0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1"} Oct 30 18:39:07.199931 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:07.199513 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:07.199931 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:07.199560 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:07.199931 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:07.199585 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:07.202207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:07.202184 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:07.208114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:07.208095 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:07.270377 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:07.270339 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=10.270326908 podStartE2EDuration="10.270326908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:07.257641788 +0000 UTC m=+2560.245982270" watchObservedRunningTime="2026-10-30 18:39:07.270326908 +0000 UTC m=+2560.258667457" Oct 30 18:39:11.368213 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.368176 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:39:11.368723 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.368481 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="fe1297f2-5abb-4173-a2c0-c47950da58ed" containerName="otc-container" containerID="cri-o://2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d" gracePeriod=30 Oct 30 18:39:11.632805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.632725 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:11.780851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.780823 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1297f2-5abb-4173-a2c0-c47950da58ed-otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe1297f2-5abb-4173-a2c0-c47950da58ed-otc-internal\") pod \"fe1297f2-5abb-4173-a2c0-c47950da58ed\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " Oct 30 18:39:11.781035 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.780874 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-monitoring-tls-certs\") pod \"fe1297f2-5abb-4173-a2c0-c47950da58ed\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " Oct 30 18:39:11.781035 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.780896 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-otlp-tls-certs\") pod \"fe1297f2-5abb-4173-a2c0-c47950da58ed\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " Oct 30 18:39:11.781035 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.780935 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1297f2-5abb-4173-a2c0-c47950da58ed-kube-api-access-4jmhc\" (UniqueName: \"kubernetes.io/projected/fe1297f2-5abb-4173-a2c0-c47950da58ed-kube-api-access-4jmhc\") pod \"fe1297f2-5abb-4173-a2c0-c47950da58ed\" (UID: \"fe1297f2-5abb-4173-a2c0-c47950da58ed\") " Oct 30 18:39:11.781229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.781198 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1297f2-5abb-4173-a2c0-c47950da58ed-otc-internal" pod "fe1297f2-5abb-4173-a2c0-c47950da58ed" (UID: "fe1297f2-5abb-4173-a2c0-c47950da58ed"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:11.782992 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.782967 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-monitoring-tls-certs" pod "fe1297f2-5abb-4173-a2c0-c47950da58ed" (UID: "fe1297f2-5abb-4173-a2c0-c47950da58ed"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:11.783119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.783029 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1297f2-5abb-4173-a2c0-c47950da58ed-kube-api-access-4jmhc" pod "fe1297f2-5abb-4173-a2c0-c47950da58ed" (UID: "fe1297f2-5abb-4173-a2c0-c47950da58ed"). InnerVolumeSpecName "kube-api-access-4jmhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:11.783119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.783037 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-otlp-tls-certs" pod "fe1297f2-5abb-4173-a2c0-c47950da58ed" (UID: "fe1297f2-5abb-4173-a2c0-c47950da58ed"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:11.882372 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.882350 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe1297f2-5abb-4173-a2c0-c47950da58ed-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.882372 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.882372 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.882514 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.882381 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1297f2-5abb-4173-a2c0-c47950da58ed-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.882514 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.882391 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jmhc\" (UniqueName: \"kubernetes.io/projected/fe1297f2-5abb-4173-a2c0-c47950da58ed-kube-api-access-4jmhc\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.913385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.913317 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:39:11.913620 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.913562 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo" containerID="cri-o://3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93" gracePeriod=30 Oct 30 18:39:11.913753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.913636 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo-gateway-opa" containerID="cri-o://3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f" gracePeriod=30 Oct 30 18:39:11.913753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:11.913744 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo-gateway" containerID="cri-o://0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1" gracePeriod=30 Oct 30 18:39:12.200112 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.200051 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.83:8083/ready\": dial tcp 10.132.0.83:8083: connect: connection refused" Oct 30 18:39:12.203809 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.203783 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:39:12.223419 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.223397 5014 generic.go:356] "Generic (PLEG): container finished" podID="764fef65-8431-4f94-9658-c637818790ec" containerID="3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f" exitCode=0 Oct 30 18:39:12.223419 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.223416 5014 generic.go:356] "Generic (PLEG): container finished" podID="764fef65-8431-4f94-9658-c637818790ec" containerID="3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93" exitCode=0 Oct 30 18:39:12.223576 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.223469 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"764fef65-8431-4f94-9658-c637818790ec","Type":"ContainerDied","Data":"3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f"} Oct 30 18:39:12.223576 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.223513 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"764fef65-8431-4f94-9658-c637818790ec","Type":"ContainerDied","Data":"3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93"} Oct 30 18:39:12.224868 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.224850 5014 generic.go:356] "Generic (PLEG): container finished" podID="fe1297f2-5abb-4173-a2c0-c47950da58ed" containerID="2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d" exitCode=0 Oct 30 18:39:12.224949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.224876 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fe1297f2-5abb-4173-a2c0-c47950da58ed","Type":"ContainerDied","Data":"2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d"} Oct 30 18:39:12.224949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.224898 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:12.224949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.224921 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fe1297f2-5abb-4173-a2c0-c47950da58ed","Type":"ContainerDied","Data":"2b8dfb4732c25cfe369f7d56c741649e4685b8489ad45c30270b846779e5a7bd"} Oct 30 18:39:12.224949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.224943 5014 scope.go:122] "RemoveContainer" containerID="2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d" Oct 30 18:39:12.240188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.240171 5014 scope.go:122] "RemoveContainer" containerID="2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d" Oct 30 18:39:12.240456 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:39:12.240437 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d\": container with ID starting with 2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d not found: ID does not exist" containerID="2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d" Oct 30 18:39:12.240508 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.240463 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d"} err="failed to get container status \"2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d\": rpc error: code = NotFound desc = could not find container \"2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d\": container with ID starting with 2649b8095e0b6e0860a55269e088e1663512487d14d9dade78a2732978826c3d not found: ID does not exist" Oct 30 18:39:12.277753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.277732 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:39:12.281843 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:12.281825 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:39:13.141999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.141975 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe1297f2-5abb-4173-a2c0-c47950da58ed" path="/var/lib/kubelet/pods/fe1297f2-5abb-4173-a2c0-c47950da58ed/volumes" Oct 30 18:39:13.161661 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.161638 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:13.231167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.231144 5014 generic.go:356] "Generic (PLEG): container finished" podID="764fef65-8431-4f94-9658-c637818790ec" containerID="0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1" exitCode=0 Oct 30 18:39:13.231290 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.231186 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:13.231290 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.231203 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"764fef65-8431-4f94-9658-c637818790ec","Type":"ContainerDied","Data":"0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1"} Oct 30 18:39:13.231290 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.231266 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"764fef65-8431-4f94-9658-c637818790ec","Type":"ContainerDied","Data":"8efc5ed41adfe8535dbc0c616d48620a86bb8424aa74fffb279aea3dd5056efe"} Oct 30 18:39:13.231459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.231299 5014 scope.go:122] "RemoveContainer" containerID="3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f" Oct 30 18:39:13.240567 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.240550 5014 scope.go:122] "RemoveContainer" containerID="0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1" Oct 30 18:39:13.248391 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.248371 5014 scope.go:122] "RemoveContainer" containerID="3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93" Oct 30 18:39:13.255972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.255952 5014 scope.go:122] "RemoveContainer" containerID="3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f" Oct 30 18:39:13.256270 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:39:13.256245 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f\": container with ID starting with 3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f not found: ID does not exist" containerID="3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f" Oct 30 18:39:13.256327 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.256269 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f"} err="failed to get container status \"3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f\": rpc error: code = NotFound desc = could not find container \"3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f\": container with ID starting with 3eb07935eeca4037f1b77850f267e188265bd51c67685d8630d743809dda461f not found: ID does not exist" Oct 30 18:39:13.256327 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.256287 5014 scope.go:122] "RemoveContainer" containerID="0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1" Oct 30 18:39:13.256534 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:39:13.256518 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1\": container with ID starting with 0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1 not found: ID does not exist" containerID="0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1" Oct 30 18:39:13.256582 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.256541 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1"} err="failed to get container status \"0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1\": rpc error: code = NotFound desc = could not find container \"0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1\": container with ID starting with 0e6ce3c80fb4f0f28fd4728554cd3534cff4ceaae697c1f471dacbe767ad7dd1 not found: ID does not exist" Oct 30 18:39:13.256582 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.256560 5014 scope.go:122] "RemoveContainer" containerID="3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93" Oct 30 18:39:13.256803 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:39:13.256785 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93\": container with ID starting with 3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93 not found: ID does not exist" containerID="3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93" Oct 30 18:39:13.256859 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.256813 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93"} err="failed to get container status \"3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93\": rpc error: code = NotFound desc = could not find container \"3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93\": container with ID starting with 3d0301751ff64ba840ff52cbaeb7e7c0463868faeea136dc14521c9602cbec93 not found: ID does not exist" Oct 30 18:39:13.293077 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.293058 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"764fef65-8431-4f94-9658-c637818790ec\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " Oct 30 18:39:13.293167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.293090 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-conf\") pod \"764fef65-8431-4f94-9658-c637818790ec\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " Oct 30 18:39:13.293167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.293111 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-mtls\") pod \"764fef65-8431-4f94-9658-c637818790ec\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " Oct 30 18:39:13.293251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.293217 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"764fef65-8431-4f94-9658-c637818790ec\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " Oct 30 18:39:13.293251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.293247 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-gateway-tenants\") pod \"764fef65-8431-4f94-9658-c637818790ec\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " Oct 30 18:39:13.293318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.293281 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"764fef65-8431-4f94-9658-c637818790ec\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " Oct 30 18:39:13.293449 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.293430 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-gateway-rbac\") pod \"764fef65-8431-4f94-9658-c637818790ec\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " Oct 30 18:39:13.293514 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.293497 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764fef65-8431-4f94-9658-c637818790ec-kube-api-access-mkd7n\" (UniqueName: \"kubernetes.io/projected/764fef65-8431-4f94-9658-c637818790ec-kube-api-access-mkd7n\") pod \"764fef65-8431-4f94-9658-c637818790ec\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " Oct 30 18:39:13.293545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.293538 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-ca-bundle\") pod \"764fef65-8431-4f94-9658-c637818790ec\" (UID: \"764fef65-8431-4f94-9658-c637818790ec\") " Oct 30 18:39:13.296089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.296032 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "764fef65-8431-4f94-9658-c637818790ec" (UID: "764fef65-8431-4f94-9658-c637818790ec"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:39:13.296360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.296153 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-conf" pod "764fef65-8431-4f94-9658-c637818790ec" (UID: "764fef65-8431-4f94-9658-c637818790ec"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:13.296360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.296225 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-serving-cabundle" pod "764fef65-8431-4f94-9658-c637818790ec" (UID: "764fef65-8431-4f94-9658-c637818790ec"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:13.296760 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.296679 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-gateway-rbac" pod "764fef65-8431-4f94-9658-c637818790ec" (UID: "764fef65-8431-4f94-9658-c637818790ec"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:13.296760 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.296705 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-ca-bundle" pod "764fef65-8431-4f94-9658-c637818790ec" (UID: "764fef65-8431-4f94-9658-c637818790ec"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:13.297784 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.297764 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "764fef65-8431-4f94-9658-c637818790ec" (UID: "764fef65-8431-4f94-9658-c637818790ec"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:13.297935 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.297920 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-gateway-tenants" pod "764fef65-8431-4f94-9658-c637818790ec" (UID: "764fef65-8431-4f94-9658-c637818790ec"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:13.298101 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.298076 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-mtls" pod "764fef65-8431-4f94-9658-c637818790ec" (UID: "764fef65-8431-4f94-9658-c637818790ec"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:13.298246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.298231 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764fef65-8431-4f94-9658-c637818790ec-kube-api-access-mkd7n" pod "764fef65-8431-4f94-9658-c637818790ec" (UID: "764fef65-8431-4f94-9658-c637818790ec"). InnerVolumeSpecName "kube-api-access-mkd7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:13.332330 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:39:13.332299 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:39:13.395010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.394987 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-gateway-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:13.395010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.395009 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkd7n\" (UniqueName: \"kubernetes.io/projected/764fef65-8431-4f94-9658-c637818790ec-kube-api-access-mkd7n\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:13.395169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.395020 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:13.395169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.395031 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:13.395169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.395040 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:13.395169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.395049 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:13.395169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.395075 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:39:13.395169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.395085 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/764fef65-8431-4f94-9658-c637818790ec-gateway-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:13.395169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.395094 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/764fef65-8431-4f94-9658-c637818790ec-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:39:13.404398 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.404383 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:39:13.496274 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.496252 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" Oct 30 18:39:13.576291 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.576266 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:39:13.588188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:13.588162 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:39:14.714627 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.714576 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.714911 5014 cpu_manager.go:414] "containerMap: removing container" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo" Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.714921 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo" Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.714935 5014 cpu_manager.go:414] "containerMap: removing container" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo-gateway-opa" Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.714940 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo-gateway-opa" Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.714953 5014 cpu_manager.go:414] "containerMap: removing container" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo-gateway" Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.714958 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo-gateway" Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.714977 5014 cpu_manager.go:414] "containerMap: removing container" podUID="fe1297f2-5abb-4173-a2c0-c47950da58ed" containerName="otc-container" Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.714981 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1297f2-5abb-4173-a2c0-c47950da58ed" containerName="otc-container" Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.715042 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo" Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.715050 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo-gateway-opa" Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.715057 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="764fef65-8431-4f94-9658-c637818790ec" containerName="tempo-gateway" Oct 30 18:39:14.715091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.715066 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1297f2-5abb-4173-a2c0-c47950da58ed" containerName="otc-container" Oct 30 18:39:14.719673 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.719652 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.743616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.742063 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:39:14.744010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.743708 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:39:14.745005 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.744980 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:39:14.753516 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.753477 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 18:39:14.753701 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.753680 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8qss4\"" Oct 30 18:39:14.753942 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.753920 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:39:14.754032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.753991 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:39:14.807103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.807079 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdwv\" (UniqueName: \"kubernetes.io/projected/363b89ca-e525-4c9c-92eb-094bd6ffe841-kube-api-access-frdwv\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.807266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.807109 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.807266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.807159 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.807397 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.807314 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/363b89ca-e525-4c9c-92eb-094bd6ffe841-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.908719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.908680 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frdwv\" (UniqueName: \"kubernetes.io/projected/363b89ca-e525-4c9c-92eb-094bd6ffe841-kube-api-access-frdwv\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.908860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.908726 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.908860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.908782 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.908860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.908828 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/363b89ca-e525-4c9c-92eb-094bd6ffe841-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.909452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.909431 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/363b89ca-e525-4c9c-92eb-094bd6ffe841-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.911358 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.911335 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.911456 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.911373 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:14.933381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:14.933359 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frdwv\" (UniqueName: \"kubernetes.io/projected/363b89ca-e525-4c9c-92eb-094bd6ffe841-kube-api-access-frdwv\") pod \"data-science-collector-collector-0\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:15.033848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.033796 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:39:15.140966 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.140939 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764fef65-8431-4f94-9658-c637818790ec" path="/var/lib/kubelet/pods/764fef65-8431-4f94-9658-c637818790ec/volumes" Oct 30 18:39:15.333236 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.333183 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:39:15.337923 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:39:15.337899 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363b89ca_e525_4c9c_92eb_094bd6ffe841.slice/crio-e29942a2219c3da78956ac2dbc61dcd23ca8e9419cd33ced27eb0215c8135b5f WatchSource:0}: Error finding container e29942a2219c3da78956ac2dbc61dcd23ca8e9419cd33ced27eb0215c8135b5f: Status 404 returned error can't find the container with id e29942a2219c3da78956ac2dbc61dcd23ca8e9419cd33ced27eb0215c8135b5f Oct 30 18:39:15.339936 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.339909 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:39:15.339986 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.339965 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:39:15.511928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.511891 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:39:15.516212 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.516189 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.532399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.532382 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:39:15.532904 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.532884 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:39:15.532972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.532891 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:39:15.532972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.532928 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:39:15.533108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.533078 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:39:15.544404 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.544264 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:39:15.544404 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.544316 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-ps5kh\"" Oct 30 18:39:15.544404 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.544362 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:39:15.546111 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.546094 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:39:15.614963 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.614897 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.614963 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.614927 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.614963 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.614948 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.615144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.614967 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqbk\" (UniqueName: \"kubernetes.io/projected/f7105f9c-a976-40fa-b693-dd7e68a9ef53-kube-api-access-fdqbk\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.615144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.615028 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.615144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.615102 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.615255 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.615147 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.615255 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.615181 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.716002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.715976 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.716002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.716004 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.716480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.716026 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.716480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.716047 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqbk\" (UniqueName: \"kubernetes.io/projected/f7105f9c-a976-40fa-b693-dd7e68a9ef53-kube-api-access-fdqbk\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.716480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.716076 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.716480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.716111 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.716480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.716149 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.716480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.716271 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.716890 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.716864 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.716890 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.716879 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.717165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.716873 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.717220 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.717193 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.718871 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.718851 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.718982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.718962 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.719038 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.718997 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:15.739048 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:15.739029 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqbk\" (UniqueName: \"kubernetes.io/projected/f7105f9c-a976-40fa-b693-dd7e68a9ef53-kube-api-access-fdqbk\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:16.255032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:16.254993 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"363b89ca-e525-4c9c-92eb-094bd6ffe841","Type":"ContainerStarted","Data":"5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c"} Oct 30 18:39:16.255183 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:16.255091 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"363b89ca-e525-4c9c-92eb-094bd6ffe841","Type":"ContainerStarted","Data":"e29942a2219c3da78956ac2dbc61dcd23ca8e9419cd33ced27eb0215c8135b5f"} Oct 30 18:39:16.296680 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:16.296641 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.296625185 podStartE2EDuration="2.296625185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:16.295717085 +0000 UTC m=+2569.284057572" watchObservedRunningTime="2026-10-30 18:39:16.296625185 +0000 UTC m=+2569.284965673" Oct 30 18:39:20.957672 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:20.957640 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:20.998029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:20.997999 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:21.058937 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:21.058912 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:21.059075 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:21.059035 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:21.073147 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:21.073127 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:21.174297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:21.174270 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:21.201290 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:21.201270 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:21.232558 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:21.232508 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:21.559948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:21.559917 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:39:21.564388 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:39:21.564359 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7105f9c_a976_40fa_b693_dd7e68a9ef53.slice/crio-937efd4878a24aa036b24c0c90c349cac92037580f6e58756d41b0e1011d8ac6 WatchSource:0}: Error finding container 937efd4878a24aa036b24c0c90c349cac92037580f6e58756d41b0e1011d8ac6: Status 404 returned error can't find the container with id 937efd4878a24aa036b24c0c90c349cac92037580f6e58756d41b0e1011d8ac6 Oct 30 18:39:22.289227 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:22.289191 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f7105f9c-a976-40fa-b693-dd7e68a9ef53","Type":"ContainerStarted","Data":"f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a"} Oct 30 18:39:22.289735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:22.289363 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:22.289735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:22.289526 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:22.289735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:22.289574 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f7105f9c-a976-40fa-b693-dd7e68a9ef53","Type":"ContainerStarted","Data":"67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d"} Oct 30 18:39:22.289735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:22.289620 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f7105f9c-a976-40fa-b693-dd7e68a9ef53","Type":"ContainerStarted","Data":"7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989"} Oct 30 18:39:22.289735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:22.289649 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f7105f9c-a976-40fa-b693-dd7e68a9ef53","Type":"ContainerStarted","Data":"937efd4878a24aa036b24c0c90c349cac92037580f6e58756d41b0e1011d8ac6"} Oct 30 18:39:22.292253 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:22.292221 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:22.301802 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:22.301782 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:22.347668 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:22.347627 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=7.347615794 podStartE2EDuration="7.347615794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:22.346777503 +0000 UTC m=+2575.335117989" watchObservedRunningTime="2026-10-30 18:39:22.347615794 +0000 UTC m=+2575.335956273" Oct 30 18:39:23.295317 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:23.295290 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:39:43.331509 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:39:43.331468 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:39:44.301922 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:44.301869 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:39:54.301656 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:39:54.301610 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:40:04.302297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:04.302255 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:40:06.464794 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.464759 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:40:06.465239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.465047 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="363b89ca-e525-4c9c-92eb-094bd6ffe841" containerName="otc-container" containerID="cri-o://5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c" gracePeriod=30 Oct 30 18:40:06.717922 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.717864 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:06.808427 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.808394 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-otlp-tls-certs\") pod \"363b89ca-e525-4c9c-92eb-094bd6ffe841\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " Oct 30 18:40:06.808575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.808488 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363b89ca-e525-4c9c-92eb-094bd6ffe841-otc-internal\" (UniqueName: \"kubernetes.io/configmap/363b89ca-e525-4c9c-92eb-094bd6ffe841-otc-internal\") pod \"363b89ca-e525-4c9c-92eb-094bd6ffe841\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " Oct 30 18:40:06.808575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.808543 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363b89ca-e525-4c9c-92eb-094bd6ffe841-kube-api-access-frdwv\" (UniqueName: \"kubernetes.io/projected/363b89ca-e525-4c9c-92eb-094bd6ffe841-kube-api-access-frdwv\") pod \"363b89ca-e525-4c9c-92eb-094bd6ffe841\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " Oct 30 18:40:06.808685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.808576 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-monitoring-tls-certs\") pod \"363b89ca-e525-4c9c-92eb-094bd6ffe841\" (UID: \"363b89ca-e525-4c9c-92eb-094bd6ffe841\") " Oct 30 18:40:06.808891 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.808864 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363b89ca-e525-4c9c-92eb-094bd6ffe841-otc-internal" pod "363b89ca-e525-4c9c-92eb-094bd6ffe841" (UID: "363b89ca-e525-4c9c-92eb-094bd6ffe841"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:06.810632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.810583 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-otlp-tls-certs" pod "363b89ca-e525-4c9c-92eb-094bd6ffe841" (UID: "363b89ca-e525-4c9c-92eb-094bd6ffe841"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:06.810818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.810801 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-monitoring-tls-certs" pod "363b89ca-e525-4c9c-92eb-094bd6ffe841" (UID: "363b89ca-e525-4c9c-92eb-094bd6ffe841"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:06.810881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.810814 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363b89ca-e525-4c9c-92eb-094bd6ffe841-kube-api-access-frdwv" pod "363b89ca-e525-4c9c-92eb-094bd6ffe841" (UID: "363b89ca-e525-4c9c-92eb-094bd6ffe841"). InnerVolumeSpecName "kube-api-access-frdwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:06.894260 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.894232 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:40:06.894525 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.894495 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo" containerID="cri-o://7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989" gracePeriod=30 Oct 30 18:40:06.894770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.894578 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo-gateway-opa" containerID="cri-o://f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a" gracePeriod=30 Oct 30 18:40:06.894770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.894706 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo-gateway" containerID="cri-o://67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d" gracePeriod=30 Oct 30 18:40:06.909763 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.909741 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:06.909863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.909766 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/363b89ca-e525-4c9c-92eb-094bd6ffe841-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:06.909863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.909780 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frdwv\" (UniqueName: \"kubernetes.io/projected/363b89ca-e525-4c9c-92eb-094bd6ffe841-kube-api-access-frdwv\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:06.909863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:06.909792 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/363b89ca-e525-4c9c-92eb-094bd6ffe841-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:07.289702 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.289626 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.85:8083/ready\": dial tcp 10.132.0.85:8083: connect: connection refused" Oct 30 18:40:07.293405 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.293370 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:40:07.490764 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.490734 5014 generic.go:356] "Generic (PLEG): container finished" podID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerID="f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a" exitCode=0 Oct 30 18:40:07.490764 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.490756 5014 generic.go:356] "Generic (PLEG): container finished" podID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerID="7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989" exitCode=0 Oct 30 18:40:07.491125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.490809 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f7105f9c-a976-40fa-b693-dd7e68a9ef53","Type":"ContainerDied","Data":"f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a"} Oct 30 18:40:07.491125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.490856 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f7105f9c-a976-40fa-b693-dd7e68a9ef53","Type":"ContainerDied","Data":"7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989"} Oct 30 18:40:07.492221 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.492199 5014 generic.go:356] "Generic (PLEG): container finished" podID="363b89ca-e525-4c9c-92eb-094bd6ffe841" containerID="5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c" exitCode=0 Oct 30 18:40:07.492316 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.492226 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"363b89ca-e525-4c9c-92eb-094bd6ffe841","Type":"ContainerDied","Data":"5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c"} Oct 30 18:40:07.492316 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.492260 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:07.492316 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.492274 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"363b89ca-e525-4c9c-92eb-094bd6ffe841","Type":"ContainerDied","Data":"e29942a2219c3da78956ac2dbc61dcd23ca8e9419cd33ced27eb0215c8135b5f"} Oct 30 18:40:07.492316 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.492299 5014 scope.go:122] "RemoveContainer" containerID="5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c" Oct 30 18:40:07.501054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.501036 5014 scope.go:122] "RemoveContainer" containerID="5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c" Oct 30 18:40:07.501329 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:40:07.501311 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c\": container with ID starting with 5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c not found: ID does not exist" containerID="5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c" Oct 30 18:40:07.501375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.501344 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c"} err="failed to get container status \"5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c\": rpc error: code = NotFound desc = could not find container \"5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c\": container with ID starting with 5702a43a50a25a5afb3e52d0f4966a859340f7da740fb4c204736a43251e2f2c not found: ID does not exist" Oct 30 18:40:07.542080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.542030 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:40:07.546099 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:07.546077 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:40:08.148028 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.148007 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:40:08.320845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.320821 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7105f9c-a976-40fa-b693-dd7e68a9ef53-kube-api-access-fdqbk\" (UniqueName: \"kubernetes.io/projected/f7105f9c-a976-40fa-b693-dd7e68a9ef53-kube-api-access-fdqbk\") pod \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " Oct 30 18:40:08.320985 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.320865 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-conf\") pod \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " Oct 30 18:40:08.320985 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.320953 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " Oct 30 18:40:08.321068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.320990 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " Oct 30 18:40:08.321068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.321009 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-mtls\") pod \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " Oct 30 18:40:08.321068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.321047 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-rbac\") pod \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " Oct 30 18:40:08.321211 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.321094 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-tenants\") pod \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " Oct 30 18:40:08.321211 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.321130 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " Oct 30 18:40:08.321211 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.321161 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-ca-bundle\") pod \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\" (UID: \"f7105f9c-a976-40fa-b693-dd7e68a9ef53\") " Oct 30 18:40:08.321366 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.321225 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-conf" pod "f7105f9c-a976-40fa-b693-dd7e68a9ef53" (UID: "f7105f9c-a976-40fa-b693-dd7e68a9ef53"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:08.321654 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.321504 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.321654 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.321578 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-rbac" pod "f7105f9c-a976-40fa-b693-dd7e68a9ef53" (UID: "f7105f9c-a976-40fa-b693-dd7e68a9ef53"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:08.321822 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.321689 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-serving-cabundle" pod "f7105f9c-a976-40fa-b693-dd7e68a9ef53" (UID: "f7105f9c-a976-40fa-b693-dd7e68a9ef53"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:08.322148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.322125 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-ca-bundle" pod "f7105f9c-a976-40fa-b693-dd7e68a9ef53" (UID: "f7105f9c-a976-40fa-b693-dd7e68a9ef53"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:08.323490 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.323460 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7105f9c-a976-40fa-b693-dd7e68a9ef53-kube-api-access-fdqbk" pod "f7105f9c-a976-40fa-b693-dd7e68a9ef53" (UID: "f7105f9c-a976-40fa-b693-dd7e68a9ef53"). InnerVolumeSpecName "kube-api-access-fdqbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:08.323638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.323532 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "f7105f9c-a976-40fa-b693-dd7e68a9ef53" (UID: "f7105f9c-a976-40fa-b693-dd7e68a9ef53"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:08.323881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.323859 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-mtls" pod "f7105f9c-a976-40fa-b693-dd7e68a9ef53" (UID: "f7105f9c-a976-40fa-b693-dd7e68a9ef53"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:08.323963 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.323904 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-tenants" pod "f7105f9c-a976-40fa-b693-dd7e68a9ef53" (UID: "f7105f9c-a976-40fa-b693-dd7e68a9ef53"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:08.324492 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.324472 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "f7105f9c-a976-40fa-b693-dd7e68a9ef53" (UID: "f7105f9c-a976-40fa-b693-dd7e68a9ef53"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:40:08.422733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.422706 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.422733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.422729 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.422894 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.422740 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.422894 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.422748 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f7105f9c-a976-40fa-b693-dd7e68a9ef53-gateway-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.422894 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.422757 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.422894 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.422766 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7105f9c-a976-40fa-b693-dd7e68a9ef53-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.422894 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.422775 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdqbk\" (UniqueName: \"kubernetes.io/projected/f7105f9c-a976-40fa-b693-dd7e68a9ef53-kube-api-access-fdqbk\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.422894 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.422801 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:40:08.432083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.432067 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:40:08.500099 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.500070 5014 generic.go:356] "Generic (PLEG): container finished" podID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerID="67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d" exitCode=0 Oct 30 18:40:08.500493 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.500122 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:40:08.500493 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.500140 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f7105f9c-a976-40fa-b693-dd7e68a9ef53","Type":"ContainerDied","Data":"67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d"} Oct 30 18:40:08.500493 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.500200 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f7105f9c-a976-40fa-b693-dd7e68a9ef53","Type":"ContainerDied","Data":"937efd4878a24aa036b24c0c90c349cac92037580f6e58756d41b0e1011d8ac6"} Oct 30 18:40:08.500493 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.500237 5014 scope.go:122] "RemoveContainer" containerID="f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a" Oct 30 18:40:08.509338 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.509323 5014 scope.go:122] "RemoveContainer" containerID="67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d" Oct 30 18:40:08.517304 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.517287 5014 scope.go:122] "RemoveContainer" containerID="7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989" Oct 30 18:40:08.523690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.523668 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" Oct 30 18:40:08.524849 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.524827 5014 scope.go:122] "RemoveContainer" containerID="f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a" Oct 30 18:40:08.525119 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:40:08.525097 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a\": container with ID starting with f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a not found: ID does not exist" containerID="f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a" Oct 30 18:40:08.525182 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.525128 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a"} err="failed to get container status \"f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a\": rpc error: code = NotFound desc = could not find container \"f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a\": container with ID starting with f5fa22a3840b61c43c405cfda0bbc8f7557f4e97d39b62e7898ca5d39dc8e67a not found: ID does not exist" Oct 30 18:40:08.525182 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.525144 5014 scope.go:122] "RemoveContainer" containerID="67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d" Oct 30 18:40:08.525371 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:40:08.525355 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d\": container with ID starting with 67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d not found: ID does not exist" containerID="67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d" Oct 30 18:40:08.525413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.525379 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d"} err="failed to get container status \"67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d\": rpc error: code = NotFound desc = could not find container \"67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d\": container with ID starting with 67008ae8cfbe1ff43d29fec778b5fd8eb04049115f81a79fc6830ecd32b4671d not found: ID does not exist" Oct 30 18:40:08.525413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.525393 5014 scope.go:122] "RemoveContainer" containerID="7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989" Oct 30 18:40:08.525646 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:40:08.525621 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989\": container with ID starting with 7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989 not found: ID does not exist" containerID="7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989" Oct 30 18:40:08.525689 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.525656 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989"} err="failed to get container status \"7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989\": rpc error: code = NotFound desc = could not find container \"7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989\": container with ID starting with 7fef8d736ed5d6103002f4ab8b28a7198735a3ee9502c0d69de6f64396704989 not found: ID does not exist" Oct 30 18:40:08.550074 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.550054 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:40:08.561566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:08.561548 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:40:09.141069 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:09.141040 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363b89ca-e525-4c9c-92eb-094bd6ffe841" path="/var/lib/kubelet/pods/363b89ca-e525-4c9c-92eb-094bd6ffe841/volumes" Oct 30 18:40:09.141462 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:09.141448 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" path="/var/lib/kubelet/pods/f7105f9c-a976-40fa-b693-dd7e68a9ef53/volumes" Oct 30 18:40:13.333467 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:40:13.333433 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:40:27.600151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:27.600120 5014 scope.go:122] "RemoveContainer" containerID="e975c4a7157d9f201a4bc78f8eb67f1acf5ee4349c70a95338ead0769ee52dc7" Oct 30 18:40:27.608981 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:27.608955 5014 scope.go:122] "RemoveContainer" containerID="6aad179135995841a24ead2b69c7f2911646243629dade3550ca2737b910caf7" Oct 30 18:40:28.399748 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.399711 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 18:40:28.400073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400057 5014 cpu_manager.go:414] "containerMap: removing container" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo-gateway" Oct 30 18:40:28.400073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400072 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo-gateway" Oct 30 18:40:28.400205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400084 5014 cpu_manager.go:414] "containerMap: removing container" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo" Oct 30 18:40:28.400205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400089 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo" Oct 30 18:40:28.400205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400107 5014 cpu_manager.go:414] "containerMap: removing container" podUID="363b89ca-e525-4c9c-92eb-094bd6ffe841" containerName="otc-container" Oct 30 18:40:28.400205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400112 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363b89ca-e525-4c9c-92eb-094bd6ffe841" containerName="otc-container" Oct 30 18:40:28.400205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400127 5014 cpu_manager.go:414] "containerMap: removing container" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo-gateway-opa" Oct 30 18:40:28.400205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400132 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo-gateway-opa" Oct 30 18:40:28.400205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400181 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="363b89ca-e525-4c9c-92eb-094bd6ffe841" containerName="otc-container" Oct 30 18:40:28.400205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400190 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo-gateway-opa" Oct 30 18:40:28.400205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400197 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo-gateway" Oct 30 18:40:28.400205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.400203 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7105f9c-a976-40fa-b693-dd7e68a9ef53" containerName="tempo" Oct 30 18:40:28.404743 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.404726 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:40:28.419808 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.419783 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 18:40:28.419958 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.419937 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:40:28.420442 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.420422 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:40:28.434453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.434428 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-5mtd6\"" Oct 30 18:40:28.472436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.472417 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhrx\" (UniqueName: \"kubernetes.io/projected/fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a-kube-api-access-ldhrx\") pod \"seaweedfs-bucket-creator\" (UID: \"fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:40:28.573025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.572994 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhrx\" (UniqueName: \"kubernetes.io/projected/fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a-kube-api-access-ldhrx\") pod \"seaweedfs-bucket-creator\" (UID: \"fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:40:28.598393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.598369 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhrx\" (UniqueName: \"kubernetes.io/projected/fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a-kube-api-access-ldhrx\") pod \"seaweedfs-bucket-creator\" (UID: \"fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:40:28.727284 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:28.727230 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:40:29.021197 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:29.021131 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 18:40:29.025573 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:40:29.025546 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe9d3c6d_cb0a_494f_a1d9_fe9f2dc9575a.slice/crio-6063e3748965fc9ce6269feb65e76f5c627b376f624efcbed3fc0bedfae47f48 WatchSource:0}: Error finding container 6063e3748965fc9ce6269feb65e76f5c627b376f624efcbed3fc0bedfae47f48: Status 404 returned error can't find the container with id 6063e3748965fc9ce6269feb65e76f5c627b376f624efcbed3fc0bedfae47f48 Oct 30 18:40:29.027497 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:29.027479 5014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:40:29.602376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:29.602334 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a","Type":"ContainerStarted","Data":"6063e3748965fc9ce6269feb65e76f5c627b376f624efcbed3fc0bedfae47f48"} Oct 30 18:40:34.628999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:34.628964 5014 generic.go:356] "Generic (PLEG): container finished" podID="fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a" containerID="2f1abcda0b0bb2994c1acc3226069843857520e9bef588f8d435faa3d859b8a6" exitCode=0 Oct 30 18:40:34.629422 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:34.629011 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a","Type":"ContainerDied","Data":"2f1abcda0b0bb2994c1acc3226069843857520e9bef588f8d435faa3d859b8a6"} Oct 30 18:40:35.765166 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:35.765137 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:40:35.838489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:35.838459 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a-kube-api-access-ldhrx\" (UniqueName: \"kubernetes.io/projected/fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a-kube-api-access-ldhrx\") pod \"fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a\" (UID: \"fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a\") " Oct 30 18:40:35.840580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:35.840556 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a-kube-api-access-ldhrx" pod "fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a" (UID: "fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a"). InnerVolumeSpecName "kube-api-access-ldhrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:35.939187 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:35.939113 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldhrx\" (UniqueName: \"kubernetes.io/projected/fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a-kube-api-access-ldhrx\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:36.638132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:36.638098 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:40:36.638304 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:36.638105 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a","Type":"ContainerDied","Data":"6063e3748965fc9ce6269feb65e76f5c627b376f624efcbed3fc0bedfae47f48"} Oct 30 18:40:36.638304 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:36.638202 5014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6063e3748965fc9ce6269feb65e76f5c627b376f624efcbed3fc0bedfae47f48" Oct 30 18:40:38.272103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.271895 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:40:38.273125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.273099 5014 cpu_manager.go:414] "containerMap: removing container" podUID="fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a" containerName="create-bucket" Oct 30 18:40:38.273125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.273120 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a" containerName="create-bucket" Oct 30 18:40:38.273272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.273198 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a" containerName="create-bucket" Oct 30 18:40:38.276231 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.276213 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.302663 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.302637 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:40:38.304884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.304861 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:40:38.305062 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.304893 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:40:38.305062 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.305033 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:40:38.305209 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.305071 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 18:40:38.312254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.312217 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:40:38.312382 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.312216 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-b6s97\"" Oct 30 18:40:38.360092 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.360059 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.360259 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.360122 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8jp\" (UniqueName: \"kubernetes.io/projected/d301dc8e-fe15-406f-a608-be7bc0d74f09-kube-api-access-js8jp\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.360259 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.360185 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d301dc8e-fe15-406f-a608-be7bc0d74f09-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.360259 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.360228 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.461339 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.461303 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8jp\" (UniqueName: \"kubernetes.io/projected/d301dc8e-fe15-406f-a608-be7bc0d74f09-kube-api-access-js8jp\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.461339 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.461337 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d301dc8e-fe15-406f-a608-be7bc0d74f09-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.461536 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.461365 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.461536 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.461407 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.461990 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.461967 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d301dc8e-fe15-406f-a608-be7bc0d74f09-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.464119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.464098 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.464198 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.464136 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.485122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.485092 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8jp\" (UniqueName: \"kubernetes.io/projected/d301dc8e-fe15-406f-a608-be7bc0d74f09-kube-api-access-js8jp\") pod \"data-science-collector-collector-0\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.596351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.596320 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:40:38.893401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.893307 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:40:38.897757 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:40:38.897723 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd301dc8e_fe15_406f_a608_be7bc0d74f09.slice/crio-be53548efa6bf4ad3536dba99f31b52ffcf07c97d5710f8b6f19257269f9d949 WatchSource:0}: Error finding container be53548efa6bf4ad3536dba99f31b52ffcf07c97d5710f8b6f19257269f9d949: Status 404 returned error can't find the container with id be53548efa6bf4ad3536dba99f31b52ffcf07c97d5710f8b6f19257269f9d949 Oct 30 18:40:38.899855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.899819 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:40:38.899953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:38.899880 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:40:39.654511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:39.654476 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d301dc8e-fe15-406f-a608-be7bc0d74f09","Type":"ContainerStarted","Data":"3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c"} Oct 30 18:40:39.654999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:39.654536 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d301dc8e-fe15-406f-a608-be7bc0d74f09","Type":"ContainerStarted","Data":"be53548efa6bf4ad3536dba99f31b52ffcf07c97d5710f8b6f19257269f9d949"} Oct 30 18:40:39.695931 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:39.695880 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.69586669 podStartE2EDuration="1.69586669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:39.695182096 +0000 UTC m=+2652.683522585" watchObservedRunningTime="2026-10-30 18:40:39.69586669 +0000 UTC m=+2652.684207177" Oct 30 18:40:41.396662 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.396632 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl"] Oct 30 18:40:41.400263 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.400243 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.429696 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.429673 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\"" Oct 30 18:40:41.429842 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.429708 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 30 18:40:41.429842 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.429752 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-x79qd\"" Oct 30 18:40:41.429955 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.429867 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\"" Oct 30 18:40:41.431819 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.431798 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl"] Oct 30 18:40:41.489848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.489819 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.490015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.489865 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.490015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.489888 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.490110 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.490033 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.490110 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.490072 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhxb\" (UniqueName: \"kubernetes.io/projected/b522fc71-c6ce-4aa6-81f7-77b8534502e5-kube-api-access-bnhxb\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.590965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.590930 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.591122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.590986 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.591122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.591010 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.591122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.591049 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.591122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.591087 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhxb\" (UniqueName: \"kubernetes.io/projected/b522fc71-c6ce-4aa6-81f7-77b8534502e5-kube-api-access-bnhxb\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.591518 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.591496 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.591751 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.591731 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.591853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.591836 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.593437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.593419 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.609653 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.609628 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk"] Oct 30 18:40:41.617714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.616690 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhxb\" (UniqueName: \"kubernetes.io/projected/b522fc71-c6ce-4aa6-81f7-77b8534502e5-kube-api-access-bnhxb\") pod \"tempo-data-science-tempostack-distributor-5c67f87585-cc9gl\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.619098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.619073 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.635566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.635519 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk"] Oct 30 18:40:41.636647 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.636622 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"s3-secret\"" Oct 30 18:40:41.636747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.636708 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\"" Oct 30 18:40:41.651545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.651488 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc"] Oct 30 18:40:41.655006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.654988 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.676822 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.676800 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc"] Oct 30 18:40:41.692165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.692144 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.692264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.692185 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmbz\" (UniqueName: \"kubernetes.io/projected/16519122-bb38-4aa4-980b-82b7edd5cdc5-kube-api-access-fgmbz\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.692318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.692268 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.692318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.692310 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.692394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.692346 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-conf\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.692451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.692437 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-774zf\" (UniqueName: \"kubernetes.io/projected/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-kube-api-access-774zf\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.692487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.692480 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.692540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.692530 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.692577 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.692560 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.692636 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.692588 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.713585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.713556 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:41.725443 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.725365 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r"] Oct 30 18:40:41.730349 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.730327 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.736546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.736528 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\"" Oct 30 18:40:41.747490 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.747464 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\"" Oct 30 18:40:41.750441 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.750406 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r"] Oct 30 18:40:41.793198 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793164 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.793355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793217 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmbz\" (UniqueName: \"kubernetes.io/projected/16519122-bb38-4aa4-980b-82b7edd5cdc5-kube-api-access-fgmbz\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.793355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793253 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.793355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793289 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.793355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793321 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.793580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793354 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcdpv\" (UniqueName: \"kubernetes.io/projected/8136e1c7-0e30-4fc0-a878-1a7402a2c130-kube-api-access-qcdpv\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.793580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793398 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.793580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793425 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-conf\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.793580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793457 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774zf\" (UniqueName: \"kubernetes.io/projected/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-kube-api-access-774zf\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.793580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793500 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.793580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793535 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.793580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793572 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.793951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793628 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.793951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793660 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.793951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.793687 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.795014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.794988 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.795331 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.795276 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.795441 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.795329 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.795565 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.795506 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.795939 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.795914 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.796408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.796385 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-conf\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.800619 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.797758 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.800619 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.797815 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.822293 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.822272 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmbz\" (UniqueName: \"kubernetes.io/projected/16519122-bb38-4aa4-980b-82b7edd5cdc5-kube-api-access-fgmbz\") pod \"tempo-data-science-tempostack-querier-5958499c6b-dvvvc\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:41.822578 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.822555 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774zf\" (UniqueName: \"kubernetes.io/projected/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-kube-api-access-774zf\") pod \"tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.894357 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.894328 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.894481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.894371 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.894481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.894396 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.894481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.894431 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcdpv\" (UniqueName: \"kubernetes.io/projected/8136e1c7-0e30-4fc0-a878-1a7402a2c130-kube-api-access-qcdpv\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.894481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.894451 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.894829 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.894799 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.895131 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.895108 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.895306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.895286 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.896813 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.896795 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.917216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.917171 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcdpv\" (UniqueName: \"kubernetes.io/projected/8136e1c7-0e30-4fc0-a878-1a7402a2c130-kube-api-access-qcdpv\") pod \"tempo-data-science-tempostack-compactor-84666c8c84-vs69r\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:41.939627 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.939582 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:41.967823 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:41.967803 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:42.043579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.043392 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl"] Oct 30 18:40:42.054534 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.054510 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:42.066938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.066901 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj"] Oct 30 18:40:42.072261 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.072244 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.092399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.092228 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 30 18:40:42.092399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.092314 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 18:40:42.094148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.093382 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 30 18:40:42.094148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.093518 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-w2vq8\"" Oct 30 18:40:42.094329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.094241 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 18:40:42.096979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.095612 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 30 18:40:42.098302 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.098278 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj"] Oct 30 18:40:42.197342 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.197255 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-rbac\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.197342 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.197294 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqdxc\" (UniqueName: \"kubernetes.io/projected/e7f38503-d961-4798-b27b-83a2172ebbd9-kube-api-access-mqdxc\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.197545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.197473 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.197639 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.197558 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-cabundle\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.197750 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.197732 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tenant\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.197838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.197762 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.197838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.197799 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-serving-certs\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.239775 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.239746 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk"] Oct 30 18:40:42.245140 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:40:42.245098 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1ff3f9_58e9_472b_bde3_f17be51ea7de.slice/crio-78840f38f66667e609e163d12c84cfcf0407fb7461a66f7ca766a63ce7f1a81e WatchSource:0}: Error finding container 78840f38f66667e609e163d12c84cfcf0407fb7461a66f7ca766a63ce7f1a81e: Status 404 returned error can't find the container with id 78840f38f66667e609e163d12c84cfcf0407fb7461a66f7ca766a63ce7f1a81e Oct 30 18:40:42.273476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.273450 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc"] Oct 30 18:40:42.282733 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:40:42.282697 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16519122_bb38_4aa4_980b_82b7edd5cdc5.slice/crio-34a9be4779ba0da4aa91f606477336850677128a8c32cf84870a718752491a1e WatchSource:0}: Error finding container 34a9be4779ba0da4aa91f606477336850677128a8c32cf84870a718752491a1e: Status 404 returned error can't find the container with id 34a9be4779ba0da4aa91f606477336850677128a8c32cf84870a718752491a1e Oct 30 18:40:42.299121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.299086 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.299225 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.299157 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-cabundle\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.299225 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.299198 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tenant\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.299326 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.299224 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.299326 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.299250 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-serving-certs\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.299326 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.299292 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-rbac\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.299326 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.299321 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqdxc\" (UniqueName: \"kubernetes.io/projected/e7f38503-d961-4798-b27b-83a2172ebbd9-kube-api-access-mqdxc\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.300619 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.300459 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-cabundle\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.300619 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.300459 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.300619 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.300518 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-rbac\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.301678 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.301653 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.302809 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.302784 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tenant\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.302880 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.302791 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-serving-certs\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.322888 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.322865 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqdxc\" (UniqueName: \"kubernetes.io/projected/e7f38503-d961-4798-b27b-83a2172ebbd9-kube-api-access-mqdxc\") pod \"tempo-data-science-tempostack-gateway-845b5fcc58-g66hj\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.402246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.402220 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r"] Oct 30 18:40:42.414205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.414185 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:42.477040 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:40:42.476962 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8136e1c7_0e30_4fc0_a878_1a7402a2c130.slice/crio-b520a7270b2706c63cd2b09b1b82ccf74a386371a0a4de23c1648240f417d43b WatchSource:0}: Error finding container b520a7270b2706c63cd2b09b1b82ccf74a386371a0a4de23c1648240f417d43b: Status 404 returned error can't find the container with id b520a7270b2706c63cd2b09b1b82ccf74a386371a0a4de23c1648240f417d43b Oct 30 18:40:42.670650 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.670613 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" event={"ID":"16519122-bb38-4aa4-980b-82b7edd5cdc5","Type":"ContainerStarted","Data":"84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e"} Oct 30 18:40:42.670821 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.670673 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" event={"ID":"16519122-bb38-4aa4-980b-82b7edd5cdc5","Type":"ContainerStarted","Data":"34a9be4779ba0da4aa91f606477336850677128a8c32cf84870a718752491a1e"} Oct 30 18:40:42.670821 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.670704 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:40:42.672231 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.672203 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" event={"ID":"b522fc71-c6ce-4aa6-81f7-77b8534502e5","Type":"ContainerStarted","Data":"6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6"} Oct 30 18:40:42.672384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.672256 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:40:42.672384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.672278 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" event={"ID":"b522fc71-c6ce-4aa6-81f7-77b8534502e5","Type":"ContainerStarted","Data":"1a5799a348aa2c5e8901618a05b154f06a65ec88e30474c9acf203e28d138676"} Oct 30 18:40:42.673640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.673617 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" event={"ID":"8136e1c7-0e30-4fc0-a878-1a7402a2c130","Type":"ContainerStarted","Data":"b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e"} Oct 30 18:40:42.673764 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.673661 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" event={"ID":"8136e1c7-0e30-4fc0-a878-1a7402a2c130","Type":"ContainerStarted","Data":"b520a7270b2706c63cd2b09b1b82ccf74a386371a0a4de23c1648240f417d43b"} Oct 30 18:40:42.673764 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.673699 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:40:42.674629 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.674587 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" event={"ID":"8d1ff3f9-58e9-472b-bde3-f17be51ea7de","Type":"ContainerStarted","Data":"78840f38f66667e609e163d12c84cfcf0407fb7461a66f7ca766a63ce7f1a81e"} Oct 30 18:40:42.714738 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.714697 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" podStartSLOduration=1.714681844 podStartE2EDuration="1.714681844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:42.712554674 +0000 UTC m=+2655.700895162" watchObservedRunningTime="2026-10-30 18:40:42.714681844 +0000 UTC m=+2655.703022330" Oct 30 18:40:42.754049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.753975 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" podStartSLOduration=1.7539632840000001 podStartE2EDuration="1.753963284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:42.753532826 +0000 UTC m=+2655.741873291" watchObservedRunningTime="2026-10-30 18:40:42.753963284 +0000 UTC m=+2655.742303770" Oct 30 18:40:42.774151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.774130 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj"] Oct 30 18:40:42.795581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:42.795540 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" podStartSLOduration=1.795526074 podStartE2EDuration="1.795526074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:42.794447841 +0000 UTC m=+2655.782788325" watchObservedRunningTime="2026-10-30 18:40:42.795526074 +0000 UTC m=+2655.783866557" Oct 30 18:40:43.332448 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:40:43.332406 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:40:43.682089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:43.682054 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" event={"ID":"e7f38503-d961-4798-b27b-83a2172ebbd9","Type":"ContainerStarted","Data":"c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db"} Oct 30 18:40:43.682474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:43.682145 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:43.682474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:43.682171 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" event={"ID":"e7f38503-d961-4798-b27b-83a2172ebbd9","Type":"ContainerStarted","Data":"3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac"} Oct 30 18:40:43.682474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:43.682340 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:43.682474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:43.682397 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" event={"ID":"e7f38503-d961-4798-b27b-83a2172ebbd9","Type":"ContainerStarted","Data":"1d49ef39da9f0f3f465f51baa7ebc86a8b96cfbf4dc016f3c3605b521b63780a"} Oct 30 18:40:43.683645 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:43.683621 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.92:8083/ready\": dial tcp 10.132.0.92:8083: connect: connection refused" Oct 30 18:40:43.683751 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:43.683624 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.132.0.92:8081/ready\": dial tcp 10.132.0.92:8081: connect: connection refused" Oct 30 18:40:43.683848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:43.683823 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" event={"ID":"8d1ff3f9-58e9-472b-bde3-f17be51ea7de","Type":"ContainerStarted","Data":"ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393"} Oct 30 18:40:43.730433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:43.730391 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" podStartSLOduration=1.730380345 podStartE2EDuration="1.730380345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:43.729912508 +0000 UTC m=+2656.718252999" watchObservedRunningTime="2026-10-30 18:40:43.730380345 +0000 UTC m=+2656.718720828" Oct 30 18:40:43.773189 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:43.773155 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" podStartSLOduration=2.773144893 podStartE2EDuration="2.773144893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:43.771915134 +0000 UTC m=+2656.760255622" watchObservedRunningTime="2026-10-30 18:40:43.773144893 +0000 UTC m=+2656.761485380" Oct 30 18:40:44.490918 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.490855 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:40:44.494713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.494693 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.509497 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.509472 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\"" Oct 30 18:40:44.517577 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.517552 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:40:44.618566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.618532 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8b47\" (UniqueName: \"kubernetes.io/projected/2717e179-c335-42d2-864a-9759944dc93a-kube-api-access-p8b47\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.618776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.618583 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.618776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.618763 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.618906 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.618798 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.688639 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.688572 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.92:8083/ready\": dial tcp 10.132.0.92:8083: connect: connection refused" Oct 30 18:40:44.688639 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.688625 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:40:44.719646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.719619 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.719754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.719655 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.719754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.719733 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8b47\" (UniqueName: \"kubernetes.io/projected/2717e179-c335-42d2-864a-9759944dc93a-kube-api-access-p8b47\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.720007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.719976 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.720698 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.720674 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.720794 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.720773 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.722413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.722388 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:44.743262 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:44.743212 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8b47\" (UniqueName: \"kubernetes.io/projected/2717e179-c335-42d2-864a-9759944dc93a-kube-api-access-p8b47\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:45.176863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:45.176831 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:45.457278 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:45.457191 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj"] Oct 30 18:40:45.775025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:45.774950 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:46.698100 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:46.698067 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerName="tempo-gateway" containerID="cri-o://3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac" gracePeriod=30 Oct 30 18:40:46.698286 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:46.698074 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerName="tempo-gateway-opa" containerID="cri-o://c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db" gracePeriod=30 Oct 30 18:40:47.704758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:47.704727 5014 generic.go:356] "Generic (PLEG): container finished" podID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerID="c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db" exitCode=0 Oct 30 18:40:47.705148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:47.704792 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" event={"ID":"e7f38503-d961-4798-b27b-83a2172ebbd9","Type":"ContainerDied","Data":"c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db"} Oct 30 18:40:47.942993 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:47.942970 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:48.050865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.050785 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-serving-certs\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-serving-certs\") pod \"e7f38503-d961-4798-b27b-83a2172ebbd9\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " Oct 30 18:40:48.050865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.050851 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-rbac\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-rbac\") pod \"e7f38503-d961-4798-b27b-83a2172ebbd9\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " Oct 30 18:40:48.051059 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.050899 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-cabundle\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-cabundle\") pod \"e7f38503-d961-4798-b27b-83a2172ebbd9\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " Oct 30 18:40:48.051059 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.050936 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f38503-d961-4798-b27b-83a2172ebbd9-kube-api-access-mqdxc\" (UniqueName: \"kubernetes.io/projected/e7f38503-d961-4798-b27b-83a2172ebbd9-kube-api-access-mqdxc\") pod \"e7f38503-d961-4798-b27b-83a2172ebbd9\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " Oct 30 18:40:48.051059 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.051011 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tenant\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tenant\") pod \"e7f38503-d961-4798-b27b-83a2172ebbd9\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " Oct 30 18:40:48.051059 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.051051 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-gateway-mtls\") pod \"e7f38503-d961-4798-b27b-83a2172ebbd9\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " Oct 30 18:40:48.051278 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.051118 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-ca-bundle\") pod \"e7f38503-d961-4798-b27b-83a2172ebbd9\" (UID: \"e7f38503-d961-4798-b27b-83a2172ebbd9\") " Oct 30 18:40:48.051278 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.051252 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-cabundle" pod "e7f38503-d961-4798-b27b-83a2172ebbd9" (UID: "e7f38503-d961-4798-b27b-83a2172ebbd9"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:48.051278 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.051254 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-rbac" pod "e7f38503-d961-4798-b27b-83a2172ebbd9" (UID: "e7f38503-d961-4798-b27b-83a2172ebbd9"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:48.051449 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.051404 5014 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:48.051449 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.051423 5014 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-cabundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:48.051621 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.051568 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-ca-bundle" pod "e7f38503-d961-4798-b27b-83a2172ebbd9" (UID: "e7f38503-d961-4798-b27b-83a2172ebbd9"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:48.053188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.053158 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-serving-certs" pod "e7f38503-d961-4798-b27b-83a2172ebbd9" (UID: "e7f38503-d961-4798-b27b-83a2172ebbd9"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:48.053790 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.053766 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f38503-d961-4798-b27b-83a2172ebbd9-kube-api-access-mqdxc" pod "e7f38503-d961-4798-b27b-83a2172ebbd9" (UID: "e7f38503-d961-4798-b27b-83a2172ebbd9"). InnerVolumeSpecName "kube-api-access-mqdxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:48.053904 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.053863 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-gateway-mtls" pod "e7f38503-d961-4798-b27b-83a2172ebbd9" (UID: "e7f38503-d961-4798-b27b-83a2172ebbd9"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:48.057722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.057702 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tenant" pod "e7f38503-d961-4798-b27b-83a2172ebbd9" (UID: "e7f38503-d961-4798-b27b-83a2172ebbd9"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:48.152774 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.152748 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqdxc\" (UniqueName: \"kubernetes.io/projected/e7f38503-d961-4798-b27b-83a2172ebbd9-kube-api-access-mqdxc\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:48.152774 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.152771 5014 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tenant\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:48.152896 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.152784 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:48.152896 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.152794 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7f38503-d961-4798-b27b-83a2172ebbd9-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:48.152896 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.152803 5014 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/e7f38503-d961-4798-b27b-83a2172ebbd9-serving-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:40:48.710813 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.710781 5014 generic.go:356] "Generic (PLEG): container finished" podID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerID="3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac" exitCode=0 Oct 30 18:40:48.711193 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.710832 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" Oct 30 18:40:48.711193 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.710866 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" event={"ID":"e7f38503-d961-4798-b27b-83a2172ebbd9","Type":"ContainerDied","Data":"3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac"} Oct 30 18:40:48.711193 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.710928 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj" event={"ID":"e7f38503-d961-4798-b27b-83a2172ebbd9","Type":"ContainerDied","Data":"1d49ef39da9f0f3f465f51baa7ebc86a8b96cfbf4dc016f3c3605b521b63780a"} Oct 30 18:40:48.711193 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.710953 5014 scope.go:122] "RemoveContainer" containerID="c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db" Oct 30 18:40:48.719906 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.719891 5014 scope.go:122] "RemoveContainer" containerID="3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac" Oct 30 18:40:48.728640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.728625 5014 scope.go:122] "RemoveContainer" containerID="c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db" Oct 30 18:40:48.728875 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:40:48.728859 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db\": container with ID starting with c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db not found: ID does not exist" containerID="c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db" Oct 30 18:40:48.728914 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.728883 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db"} err="failed to get container status \"c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db\": rpc error: code = NotFound desc = could not find container \"c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db\": container with ID starting with c59e7239e6944214264664303a1cd9c321b019943ad5bd8160fb1b8ff2c097db not found: ID does not exist" Oct 30 18:40:48.728914 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.728899 5014 scope.go:122] "RemoveContainer" containerID="3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac" Oct 30 18:40:48.729114 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:40:48.729100 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac\": container with ID starting with 3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac not found: ID does not exist" containerID="3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac" Oct 30 18:40:48.729155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.729118 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac"} err="failed to get container status \"3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac\": rpc error: code = NotFound desc = could not find container \"3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac\": container with ID starting with 3e20bb1ba637d2c66f66aff5c55b7fd325f45bea246293877be4902d5a2b9aac not found: ID does not exist" Oct 30 18:40:48.773807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.773784 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj"] Oct 30 18:40:48.785640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:48.785621 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-845b5fcc58-g66hj"] Oct 30 18:40:49.141712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:49.141677 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" path="/var/lib/kubelet/pods/e7f38503-d961-4798-b27b-83a2172ebbd9/volumes" Oct 30 18:40:53.192812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:53.192784 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:53.220715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:53.220688 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:53.293469 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:53.293440 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:53.293568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:53.293538 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:53.307523 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:53.307500 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") DevicePath \"csi-2de5aa6ecd623838765e0f31befbb65b2679c3ca7e322e1ba1051ac031e970ac\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:53.534249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:53.534217 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a4e73d0eb00dee7b83c74b37dc76bd22c1b434e850c00b254cd96b669149949b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:53.558432 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:53.558407 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:53.807607 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:53.807572 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:54.122041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:54.121659 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:40:54.126026 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:40:54.126003 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2717e179_c335_42d2_864a_9759944dc93a.slice/crio-f1636da1377175dcce8bb4107fe5e2cbbe3ac0645d8ae0975bfacac2db29e6ca WatchSource:0}: Error finding container f1636da1377175dcce8bb4107fe5e2cbbe3ac0645d8ae0975bfacac2db29e6ca: Status 404 returned error can't find the container with id f1636da1377175dcce8bb4107fe5e2cbbe3ac0645d8ae0975bfacac2db29e6ca Oct 30 18:40:54.740537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:54.740499 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"2717e179-c335-42d2-864a-9759944dc93a","Type":"ContainerStarted","Data":"fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb"} Oct 30 18:40:54.741010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:54.740560 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"2717e179-c335-42d2-864a-9759944dc93a","Type":"ContainerStarted","Data":"f1636da1377175dcce8bb4107fe5e2cbbe3ac0645d8ae0975bfacac2db29e6ca"} Oct 30 18:40:54.741010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:54.740710 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:40:54.783330 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:40:54.783293 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podStartSLOduration=13.783278408 podStartE2EDuration="13.783278408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:54.782668032 +0000 UTC m=+2667.771008520" watchObservedRunningTime="2026-10-30 18:40:54.783278408 +0000 UTC m=+2667.771618906" Oct 30 18:41:03.690121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:03.690086 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:41:03.690646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:03.690413 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" podUID="8136e1c7-0e30-4fc0-a878-1a7402a2c130" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:41:03.690646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:03.690461 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:41:05.876380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:05.876349 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:41:13.332612 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:41:13.332561 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:41:13.689044 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:13.688962 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" podUID="8136e1c7-0e30-4fc0-a878-1a7402a2c130" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:41:15.750920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:15.750885 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="2717e179-c335-42d2-864a-9759944dc93a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:41:23.688876 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:23.688840 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" podUID="8136e1c7-0e30-4fc0-a878-1a7402a2c130" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:41:25.750078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:25.750036 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="2717e179-c335-42d2-864a-9759944dc93a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:41:27.387719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:27.387588 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:41:27.395116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:27.395097 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:41:33.688345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:33.688299 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" podUID="8136e1c7-0e30-4fc0-a878-1a7402a2c130" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:41:35.750153 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:35.750125 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:41:43.333800 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:41:43.333761 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:41:43.688029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:43.687944 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:41:48.664645 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.664235 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:41:48.665278 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.665233 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d301dc8e-fe15-406f-a608-be7bc0d74f09" containerName="otc-container" containerID="cri-o://3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c" gracePeriod=30 Oct 30 18:41:48.914609 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.914576 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:48.916691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.916648 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d301dc8e-fe15-406f-a608-be7bc0d74f09-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d301dc8e-fe15-406f-a608-be7bc0d74f09-otc-internal\") pod \"d301dc8e-fe15-406f-a608-be7bc0d74f09\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " Oct 30 18:41:48.916691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.916682 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-otlp-tls-certs\") pod \"d301dc8e-fe15-406f-a608-be7bc0d74f09\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " Oct 30 18:41:48.916808 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.916715 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-monitoring-tls-certs\") pod \"d301dc8e-fe15-406f-a608-be7bc0d74f09\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " Oct 30 18:41:48.916808 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.916789 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d301dc8e-fe15-406f-a608-be7bc0d74f09-kube-api-access-js8jp\" (UniqueName: \"kubernetes.io/projected/d301dc8e-fe15-406f-a608-be7bc0d74f09-kube-api-access-js8jp\") pod \"d301dc8e-fe15-406f-a608-be7bc0d74f09\" (UID: \"d301dc8e-fe15-406f-a608-be7bc0d74f09\") " Oct 30 18:41:48.917051 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.917027 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d301dc8e-fe15-406f-a608-be7bc0d74f09-otc-internal" pod "d301dc8e-fe15-406f-a608-be7bc0d74f09" (UID: "d301dc8e-fe15-406f-a608-be7bc0d74f09"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:48.918924 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.918899 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d301dc8e-fe15-406f-a608-be7bc0d74f09-kube-api-access-js8jp" pod "d301dc8e-fe15-406f-a608-be7bc0d74f09" (UID: "d301dc8e-fe15-406f-a608-be7bc0d74f09"). InnerVolumeSpecName "kube-api-access-js8jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:48.918924 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.918899 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-otlp-tls-certs" pod "d301dc8e-fe15-406f-a608-be7bc0d74f09" (UID: "d301dc8e-fe15-406f-a608-be7bc0d74f09"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:48.919036 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.918930 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-monitoring-tls-certs" pod "d301dc8e-fe15-406f-a608-be7bc0d74f09" (UID: "d301dc8e-fe15-406f-a608-be7bc0d74f09"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:48.994922 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.994887 5014 generic.go:356] "Generic (PLEG): container finished" podID="d301dc8e-fe15-406f-a608-be7bc0d74f09" containerID="3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c" exitCode=0 Oct 30 18:41:48.995043 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.994951 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:48.995043 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.994971 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d301dc8e-fe15-406f-a608-be7bc0d74f09","Type":"ContainerDied","Data":"3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c"} Oct 30 18:41:48.995043 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.995029 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d301dc8e-fe15-406f-a608-be7bc0d74f09","Type":"ContainerDied","Data":"be53548efa6bf4ad3536dba99f31b52ffcf07c97d5710f8b6f19257269f9d949"} Oct 30 18:41:48.995173 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:48.995056 5014 scope.go:122] "RemoveContainer" containerID="3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c" Oct 30 18:41:49.004551 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.004534 5014 scope.go:122] "RemoveContainer" containerID="3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c" Oct 30 18:41:49.004826 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:41:49.004806 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c\": container with ID starting with 3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c not found: ID does not exist" containerID="3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c" Oct 30 18:41:49.004884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.004834 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c"} err="failed to get container status \"3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c\": rpc error: code = NotFound desc = could not find container \"3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c\": container with ID starting with 3f5fb988c93306ea0217a448ae9d270d1a05e6eb9fd3f44d105a17f9edc0555c not found: ID does not exist" Oct 30 18:41:49.018301 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.018279 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js8jp\" (UniqueName: \"kubernetes.io/projected/d301dc8e-fe15-406f-a608-be7bc0d74f09-kube-api-access-js8jp\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:49.018301 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.018301 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d301dc8e-fe15-406f-a608-be7bc0d74f09-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:49.018403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.018311 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:49.018403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.018320 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d301dc8e-fe15-406f-a608-be7bc0d74f09-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:49.043529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.043499 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:41:49.047745 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.047728 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:41:49.083820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.083796 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:41:49.084136 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.084124 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d301dc8e-fe15-406f-a608-be7bc0d74f09" containerName="otc-container" Oct 30 18:41:49.084181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.084137 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d301dc8e-fe15-406f-a608-be7bc0d74f09" containerName="otc-container" Oct 30 18:41:49.084181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.084148 5014 cpu_manager.go:414] "containerMap: removing container" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerName="tempo-gateway-opa" Oct 30 18:41:49.084181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.084153 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerName="tempo-gateway-opa" Oct 30 18:41:49.084276 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.084185 5014 cpu_manager.go:414] "containerMap: removing container" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerName="tempo-gateway" Oct 30 18:41:49.084276 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.084191 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerName="tempo-gateway" Oct 30 18:41:49.084276 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.084250 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d301dc8e-fe15-406f-a608-be7bc0d74f09" containerName="otc-container" Oct 30 18:41:49.084276 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.084257 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerName="tempo-gateway-opa" Oct 30 18:41:49.084276 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.084266 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f38503-d961-4798-b27b-83a2172ebbd9" containerName="tempo-gateway" Oct 30 18:41:49.088659 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.088641 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.103018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.102998 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-b6s97\"" Oct 30 18:41:49.103114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.103001 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:41:49.103114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.103028 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:41:49.103114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.103057 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 18:41:49.108242 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.108223 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:41:49.118989 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.118965 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/394e9329-47b8-4454-a1d0-c79f386b5cf1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.119108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.119014 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.119203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.119169 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84v7h\" (UniqueName: \"kubernetes.io/projected/394e9329-47b8-4454-a1d0-c79f386b5cf1-kube-api-access-84v7h\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.119271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.119212 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.142468 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.142448 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d301dc8e-fe15-406f-a608-be7bc0d74f09" path="/var/lib/kubelet/pods/d301dc8e-fe15-406f-a608-be7bc0d74f09/volumes" Oct 30 18:41:49.219869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.219809 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/394e9329-47b8-4454-a1d0-c79f386b5cf1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.219970 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.219870 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.219970 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.219903 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84v7h\" (UniqueName: \"kubernetes.io/projected/394e9329-47b8-4454-a1d0-c79f386b5cf1-kube-api-access-84v7h\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.219970 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.219934 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.220443 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.220424 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/394e9329-47b8-4454-a1d0-c79f386b5cf1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.222422 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.222400 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.222503 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.222431 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.242782 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.242765 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84v7h\" (UniqueName: \"kubernetes.io/projected/394e9329-47b8-4454-a1d0-c79f386b5cf1-kube-api-access-84v7h\") pod \"data-science-collector-collector-0\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.401807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.401786 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:49.695607 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.695570 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:41:49.700306 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:41:49.700283 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394e9329_47b8_4454_a1d0_c79f386b5cf1.slice/crio-d14362f79b8f0eb45c8737afdf3668b1312cfb9ab2c5b41ceb11ace7874c16f2 WatchSource:0}: Error finding container d14362f79b8f0eb45c8737afdf3668b1312cfb9ab2c5b41ceb11ace7874c16f2: Status 404 returned error can't find the container with id d14362f79b8f0eb45c8737afdf3668b1312cfb9ab2c5b41ceb11ace7874c16f2 Oct 30 18:41:49.702577 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.702541 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:41:49.702661 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:49.702642 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:41:50.002271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:50.002190 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"394e9329-47b8-4454-a1d0-c79f386b5cf1","Type":"ContainerStarted","Data":"81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877"} Oct 30 18:41:50.002271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:50.002243 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"394e9329-47b8-4454-a1d0-c79f386b5cf1","Type":"ContainerStarted","Data":"d14362f79b8f0eb45c8737afdf3668b1312cfb9ab2c5b41ceb11ace7874c16f2"} Oct 30 18:41:50.044227 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:50.044179 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.044166546 podStartE2EDuration="1.044166546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:50.042374403 +0000 UTC m=+2723.030714891" watchObservedRunningTime="2026-10-30 18:41:50.044166546 +0000 UTC m=+2723.032507034" Oct 30 18:41:51.159096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:51.159058 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:41:51.739047 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:51.739013 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:41:51.739277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:51.739250 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="2717e179-c335-42d2-864a-9759944dc93a" containerName="tempo" containerID="cri-o://fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb" gracePeriod=30 Oct 30 18:41:51.753867 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:51.753840 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc"] Oct 30 18:41:51.754160 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:51.754094 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" podUID="16519122-bb38-4aa4-980b-82b7edd5cdc5" containerName="tempo" containerID="cri-o://84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e" gracePeriod=30 Oct 30 18:41:51.757265 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:51.757242 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl"] Oct 30 18:41:51.757778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:51.757736 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" podUID="b522fc71-c6ce-4aa6-81f7-77b8534502e5" containerName="tempo" containerID="cri-o://6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6" gracePeriod=30 Oct 30 18:41:51.765727 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:51.765700 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk"] Oct 30 18:41:51.765928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:51.765907 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" podUID="8d1ff3f9-58e9-472b-bde3-f17be51ea7de" containerName="tempo" containerID="cri-o://ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393" gracePeriod=30 Oct 30 18:41:51.766789 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:51.766762 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r"] Oct 30 18:41:51.766941 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:51.766921 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" podUID="8136e1c7-0e30-4fc0-a878-1a7402a2c130" containerName="tempo" containerID="cri-o://b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e" gracePeriod=30 Oct 30 18:41:52.011227 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.011152 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="394e9329-47b8-4454-a1d0-c79f386b5cf1" containerName="otc-container" containerID="cri-o://81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877" gracePeriod=30 Oct 30 18:41:52.263436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.263388 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:52.348881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.348854 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394e9329-47b8-4454-a1d0-c79f386b5cf1-kube-api-access-84v7h\" (UniqueName: \"kubernetes.io/projected/394e9329-47b8-4454-a1d0-c79f386b5cf1-kube-api-access-84v7h\") pod \"394e9329-47b8-4454-a1d0-c79f386b5cf1\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " Oct 30 18:41:52.349000 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.348944 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394e9329-47b8-4454-a1d0-c79f386b5cf1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/394e9329-47b8-4454-a1d0-c79f386b5cf1-otc-internal\") pod \"394e9329-47b8-4454-a1d0-c79f386b5cf1\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " Oct 30 18:41:52.349000 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.348975 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-monitoring-tls-certs\") pod \"394e9329-47b8-4454-a1d0-c79f386b5cf1\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " Oct 30 18:41:52.349081 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.349013 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-otlp-tls-certs\") pod \"394e9329-47b8-4454-a1d0-c79f386b5cf1\" (UID: \"394e9329-47b8-4454-a1d0-c79f386b5cf1\") " Oct 30 18:41:52.349313 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.349295 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394e9329-47b8-4454-a1d0-c79f386b5cf1-otc-internal" pod "394e9329-47b8-4454-a1d0-c79f386b5cf1" (UID: "394e9329-47b8-4454-a1d0-c79f386b5cf1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:52.351085 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.351055 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394e9329-47b8-4454-a1d0-c79f386b5cf1-kube-api-access-84v7h" pod "394e9329-47b8-4454-a1d0-c79f386b5cf1" (UID: "394e9329-47b8-4454-a1d0-c79f386b5cf1"). InnerVolumeSpecName "kube-api-access-84v7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:52.351185 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.351091 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-monitoring-tls-certs" pod "394e9329-47b8-4454-a1d0-c79f386b5cf1" (UID: "394e9329-47b8-4454-a1d0-c79f386b5cf1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:52.351185 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.351107 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-otlp-tls-certs" pod "394e9329-47b8-4454-a1d0-c79f386b5cf1" (UID: "394e9329-47b8-4454-a1d0-c79f386b5cf1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:52.449972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.449944 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/394e9329-47b8-4454-a1d0-c79f386b5cf1-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.449972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.449972 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.450117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.449982 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/394e9329-47b8-4454-a1d0-c79f386b5cf1-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.450117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.449991 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84v7h\" (UniqueName: \"kubernetes.io/projected/394e9329-47b8-4454-a1d0-c79f386b5cf1-kube-api-access-84v7h\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.509722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.509699 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:41:52.550366 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.550311 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-ca-bundle\") pod \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " Oct 30 18:41:52.550437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.550386 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-conf\") pod \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " Oct 30 18:41:52.550488 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.550454 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-tmp-storage\") pod \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " Oct 30 18:41:52.550549 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.550505 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-distributor-mtls\") pod \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " Oct 30 18:41:52.550549 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.550533 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b522fc71-c6ce-4aa6-81f7-77b8534502e5-kube-api-access-bnhxb\" (UniqueName: \"kubernetes.io/projected/b522fc71-c6ce-4aa6-81f7-77b8534502e5-kube-api-access-bnhxb\") pod \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\" (UID: \"b522fc71-c6ce-4aa6-81f7-77b8534502e5\") " Oct 30 18:41:52.550735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.550706 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-tmp-storage" pod "b522fc71-c6ce-4aa6-81f7-77b8534502e5" (UID: "b522fc71-c6ce-4aa6-81f7-77b8534502e5"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:52.550824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.550761 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-ca-bundle" pod "b522fc71-c6ce-4aa6-81f7-77b8534502e5" (UID: "b522fc71-c6ce-4aa6-81f7-77b8534502e5"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:52.550895 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.550876 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.550895 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.550893 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-tmp-storage\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.551140 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.551115 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-conf" pod "b522fc71-c6ce-4aa6-81f7-77b8534502e5" (UID: "b522fc71-c6ce-4aa6-81f7-77b8534502e5"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:52.553256 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.553228 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-distributor-mtls" pod "b522fc71-c6ce-4aa6-81f7-77b8534502e5" (UID: "b522fc71-c6ce-4aa6-81f7-77b8534502e5"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:52.553385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.553358 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b522fc71-c6ce-4aa6-81f7-77b8534502e5-kube-api-access-bnhxb" pod "b522fc71-c6ce-4aa6-81f7-77b8534502e5" (UID: "b522fc71-c6ce-4aa6-81f7-77b8534502e5"). InnerVolumeSpecName "kube-api-access-bnhxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:52.651996 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.651969 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.651996 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.651992 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/b522fc71-c6ce-4aa6-81f7-77b8534502e5-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.651996 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:52.652002 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnhxb\" (UniqueName: \"kubernetes.io/projected/b522fc71-c6ce-4aa6-81f7-77b8534502e5-kube-api-access-bnhxb\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.016936 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.016905 5014 generic.go:356] "Generic (PLEG): container finished" podID="b522fc71-c6ce-4aa6-81f7-77b8534502e5" containerID="6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6" exitCode=0 Oct 30 18:41:53.017071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.016952 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" Oct 30 18:41:53.017071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.016985 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" event={"ID":"b522fc71-c6ce-4aa6-81f7-77b8534502e5","Type":"ContainerDied","Data":"6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6"} Oct 30 18:41:53.017071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.017039 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl" event={"ID":"b522fc71-c6ce-4aa6-81f7-77b8534502e5","Type":"ContainerDied","Data":"1a5799a348aa2c5e8901618a05b154f06a65ec88e30474c9acf203e28d138676"} Oct 30 18:41:53.017071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.017071 5014 scope.go:122] "RemoveContainer" containerID="6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6" Oct 30 18:41:53.018329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.018314 5014 generic.go:356] "Generic (PLEG): container finished" podID="394e9329-47b8-4454-a1d0-c79f386b5cf1" containerID="81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877" exitCode=0 Oct 30 18:41:53.018403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.018368 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:41:53.018403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.018369 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"394e9329-47b8-4454-a1d0-c79f386b5cf1","Type":"ContainerDied","Data":"81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877"} Oct 30 18:41:53.018498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.018431 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"394e9329-47b8-4454-a1d0-c79f386b5cf1","Type":"ContainerDied","Data":"d14362f79b8f0eb45c8737afdf3668b1312cfb9ab2c5b41ceb11ace7874c16f2"} Oct 30 18:41:53.080237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.080212 5014 scope.go:122] "RemoveContainer" containerID="6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6" Oct 30 18:41:53.081761 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:41:53.081736 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6\": container with ID starting with 6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6 not found: ID does not exist" containerID="6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6" Oct 30 18:41:53.081890 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.081771 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6"} err="failed to get container status \"6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6\": rpc error: code = NotFound desc = could not find container \"6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6\": container with ID starting with 6b4690394c49f8a01baac8c2420a0df891eee61298b8986fa518b11ee7890ff6 not found: ID does not exist" Oct 30 18:41:53.081890 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.081797 5014 scope.go:122] "RemoveContainer" containerID="81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877" Oct 30 18:41:53.090585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.090565 5014 scope.go:122] "RemoveContainer" containerID="81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877" Oct 30 18:41:53.090896 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:41:53.090877 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877\": container with ID starting with 81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877 not found: ID does not exist" containerID="81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877" Oct 30 18:41:53.090995 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.090903 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877"} err="failed to get container status \"81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877\": rpc error: code = NotFound desc = could not find container \"81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877\": container with ID starting with 81966023ce5a534be306be0753840d95f0ff3a110024d1c0383ddbe5d6050877 not found: ID does not exist" Oct 30 18:41:53.091711 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.091690 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:41:53.102121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.102098 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:41:53.127510 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.127486 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl"] Oct 30 18:41:53.131824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.131801 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5c67f87585-cc9gl"] Oct 30 18:41:53.141353 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.141329 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394e9329-47b8-4454-a1d0-c79f386b5cf1" path="/var/lib/kubelet/pods/394e9329-47b8-4454-a1d0-c79f386b5cf1/volumes" Oct 30 18:41:53.141879 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.141860 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b522fc71-c6ce-4aa6-81f7-77b8534502e5" path="/var/lib/kubelet/pods/b522fc71-c6ce-4aa6-81f7-77b8534502e5/volumes" Oct 30 18:41:53.225858 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.225837 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:41:53.255838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.255785 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-conf\") pod \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " Oct 30 18:41:53.255838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.255830 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-tmp-storage\") pod \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " Oct 30 18:41:53.255986 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.255865 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8136e1c7-0e30-4fc0-a878-1a7402a2c130-kube-api-access-qcdpv\" (UniqueName: \"kubernetes.io/projected/8136e1c7-0e30-4fc0-a878-1a7402a2c130-kube-api-access-qcdpv\") pod \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " Oct 30 18:41:53.255986 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.255922 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-compactor-mtls\") pod \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " Oct 30 18:41:53.256100 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.255990 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-ca-bundle\") pod \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\" (UID: \"8136e1c7-0e30-4fc0-a878-1a7402a2c130\") " Oct 30 18:41:53.256168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.256145 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-tmp-storage" pod "8136e1c7-0e30-4fc0-a878-1a7402a2c130" (UID: "8136e1c7-0e30-4fc0-a878-1a7402a2c130"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:53.256215 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.256191 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-conf" pod "8136e1c7-0e30-4fc0-a878-1a7402a2c130" (UID: "8136e1c7-0e30-4fc0-a878-1a7402a2c130"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:53.256393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.256372 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.256497 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.256402 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-tmp-storage\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.256497 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.256377 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-ca-bundle" pod "8136e1c7-0e30-4fc0-a878-1a7402a2c130" (UID: "8136e1c7-0e30-4fc0-a878-1a7402a2c130"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:53.258125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.258104 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-compactor-mtls" pod "8136e1c7-0e30-4fc0-a878-1a7402a2c130" (UID: "8136e1c7-0e30-4fc0-a878-1a7402a2c130"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:53.258201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.258181 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8136e1c7-0e30-4fc0-a878-1a7402a2c130-kube-api-access-qcdpv" pod "8136e1c7-0e30-4fc0-a878-1a7402a2c130" (UID: "8136e1c7-0e30-4fc0-a878-1a7402a2c130"). InnerVolumeSpecName "kube-api-access-qcdpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:53.357195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.357168 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcdpv\" (UniqueName: \"kubernetes.io/projected/8136e1c7-0e30-4fc0-a878-1a7402a2c130-kube-api-access-qcdpv\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.357195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.357195 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.357582 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.357206 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8136e1c7-0e30-4fc0-a878-1a7402a2c130-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.637306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.637275 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 18:41:53.643573 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.643543 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 18:41:53.687691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:53.687662 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" podUID="16519122-bb38-4aa4-980b-82b7edd5cdc5" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:41:54.025477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:54.025400 5014 generic.go:356] "Generic (PLEG): container finished" podID="8136e1c7-0e30-4fc0-a878-1a7402a2c130" containerID="b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e" exitCode=0 Oct 30 18:41:54.025477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:54.025459 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" Oct 30 18:41:54.025686 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:54.025484 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" event={"ID":"8136e1c7-0e30-4fc0-a878-1a7402a2c130","Type":"ContainerDied","Data":"b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e"} Oct 30 18:41:54.025686 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:54.025565 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r" event={"ID":"8136e1c7-0e30-4fc0-a878-1a7402a2c130","Type":"ContainerDied","Data":"b520a7270b2706c63cd2b09b1b82ccf74a386371a0a4de23c1648240f417d43b"} Oct 30 18:41:54.025686 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:54.025617 5014 scope.go:122] "RemoveContainer" containerID="b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e" Oct 30 18:41:54.035062 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:54.035048 5014 scope.go:122] "RemoveContainer" containerID="b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e" Oct 30 18:41:54.035319 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:41:54.035298 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e\": container with ID starting with b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e not found: ID does not exist" containerID="b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e" Oct 30 18:41:54.035420 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:54.035322 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e"} err="failed to get container status \"b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e\": rpc error: code = NotFound desc = could not find container \"b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e\": container with ID starting with b424291c0323ad6ddd7d356c0f4945b0596efd9c44fcd750a3ddd78d8a87786e not found: ID does not exist" Oct 30 18:41:54.075285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:54.075256 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r"] Oct 30 18:41:54.080008 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:54.079987 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-84666c8c84-vs69r"] Oct 30 18:41:55.141966 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:55.141920 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8136e1c7-0e30-4fc0-a878-1a7402a2c130" path="/var/lib/kubelet/pods/8136e1c7-0e30-4fc0-a878-1a7402a2c130/volumes" Oct 30 18:41:55.142490 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:55.142385 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a" path="/var/lib/kubelet/pods/fe9d3c6d-cb0a-494f-a1d9-fe9f2dc9575a/volumes" Oct 30 18:41:55.698344 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:55.698312 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" podUID="8d1ff3f9-58e9-472b-bde3-f17be51ea7de" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:41:55.749934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:41:55.749908 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="2717e179-c335-42d2-864a-9759944dc93a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:42:03.688392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:03.688357 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" podUID="16519122-bb38-4aa4-980b-82b7edd5cdc5" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:42:05.749949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:05.749910 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="2717e179-c335-42d2-864a-9759944dc93a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:42:05.777607 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:05.777555 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" podUID="8d1ff3f9-58e9-472b-bde3-f17be51ea7de" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:42:12.678751 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.678727 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:12.801050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.800995 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ca-bundle\") pod \"2717e179-c335-42d2-864a-9759944dc93a\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " Oct 30 18:42:12.801050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.801033 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2717e179-c335-42d2-864a-9759944dc93a-kube-api-access-p8b47\" (UniqueName: \"kubernetes.io/projected/2717e179-c335-42d2-864a-9759944dc93a-kube-api-access-p8b47\") pod \"2717e179-c335-42d2-864a-9759944dc93a\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " Oct 30 18:42:12.801209 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.801080 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ingester-mtls\") pod \"2717e179-c335-42d2-864a-9759944dc93a\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " Oct 30 18:42:12.801209 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.801142 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-conf\") pod \"2717e179-c335-42d2-864a-9759944dc93a\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " Oct 30 18:42:12.801322 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.801240 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"2717e179-c335-42d2-864a-9759944dc93a\" (UID: \"2717e179-c335-42d2-864a-9759944dc93a\") " Oct 30 18:42:12.801426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.801393 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ca-bundle" pod "2717e179-c335-42d2-864a-9759944dc93a" (UID: "2717e179-c335-42d2-864a-9759944dc93a"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:12.801556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.801539 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:12.801658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.801539 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-conf" pod "2717e179-c335-42d2-864a-9759944dc93a" (UID: "2717e179-c335-42d2-864a-9759944dc93a"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:12.803416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.803396 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ingester-mtls" pod "2717e179-c335-42d2-864a-9759944dc93a" (UID: "2717e179-c335-42d2-864a-9759944dc93a"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:12.803508 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.803482 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2717e179-c335-42d2-864a-9759944dc93a-kube-api-access-p8b47" pod "2717e179-c335-42d2-864a-9759944dc93a" (UID: "2717e179-c335-42d2-864a-9759944dc93a"). InnerVolumeSpecName "kube-api-access-p8b47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:12.804125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.804108 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1" pod "2717e179-c335-42d2-864a-9759944dc93a" (UID: "2717e179-c335-42d2-864a-9759944dc93a"). InnerVolumeSpecName "pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:42:12.901854 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.901828 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8b47\" (UniqueName: \"kubernetes.io/projected/2717e179-c335-42d2-864a-9759944dc93a-kube-api-access-p8b47\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:12.901958 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.901854 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/2717e179-c335-42d2-864a-9759944dc93a-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:12.901958 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.901869 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2717e179-c335-42d2-864a-9759944dc93a-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:12.901958 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.901902 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:42:12.911248 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:12.911232 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:42:13.002863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.002840 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-2de5aa6ecd623838765e0f31befbb65b2679c3ca7e322e1ba1051ac031e970ac\"" Oct 30 18:42:13.118633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.118580 5014 generic.go:356] "Generic (PLEG): container finished" podID="2717e179-c335-42d2-864a-9759944dc93a" containerID="fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb" exitCode=0 Oct 30 18:42:13.118758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.118638 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"2717e179-c335-42d2-864a-9759944dc93a","Type":"ContainerDied","Data":"fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb"} Oct 30 18:42:13.118758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.118656 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:13.118758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.118697 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"2717e179-c335-42d2-864a-9759944dc93a","Type":"ContainerDied","Data":"f1636da1377175dcce8bb4107fe5e2cbbe3ac0645d8ae0975bfacac2db29e6ca"} Oct 30 18:42:13.118758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.118726 5014 scope.go:122] "RemoveContainer" containerID="fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb" Oct 30 18:42:13.129055 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.129037 5014 scope.go:122] "RemoveContainer" containerID="fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb" Oct 30 18:42:13.129323 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:42:13.129300 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb\": container with ID starting with fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb not found: ID does not exist" containerID="fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb" Oct 30 18:42:13.129362 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.129332 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb"} err="failed to get container status \"fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb\": rpc error: code = NotFound desc = could not find container \"fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb\": container with ID starting with fafea46ffd8939546b8bd89271579d74142cc5558f909a852a13b2a3b6bf5cbb not found: ID does not exist" Oct 30 18:42:13.169183 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.169162 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:42:13.180898 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.180879 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:42:13.333132 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:42:13.333101 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:42:13.687788 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.687753 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" podUID="16519122-bb38-4aa4-980b-82b7edd5cdc5" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:42:13.688103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:13.687853 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:42:15.141502 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:15.141461 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2717e179-c335-42d2-864a-9759944dc93a" path="/var/lib/kubelet/pods/2717e179-c335-42d2-864a-9759944dc93a/volumes" Oct 30 18:42:15.700731 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:15.700691 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" podUID="8d1ff3f9-58e9-472b-bde3-f17be51ea7de" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:42:15.700893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:15.700814 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:42:21.930243 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:21.930208 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:42:21.933086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:21.933068 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:42:22.069982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.069944 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-query-frontend-mtls\") pod \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " Oct 30 18:42:22.070148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.069992 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16519122-bb38-4aa4-980b-82b7edd5cdc5-kube-api-access-fgmbz\" (UniqueName: \"kubernetes.io/projected/16519122-bb38-4aa4-980b-82b7edd5cdc5-kube-api-access-fgmbz\") pod \"16519122-bb38-4aa4-980b-82b7edd5cdc5\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " Oct 30 18:42:22.070148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070016 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-tmp-storage\") pod \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " Oct 30 18:42:22.070148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070058 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-querier-mtls\") pod \"16519122-bb38-4aa4-980b-82b7edd5cdc5\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " Oct 30 18:42:22.070148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070091 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-tmp-storage\") pod \"16519122-bb38-4aa4-980b-82b7edd5cdc5\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " Oct 30 18:42:22.070148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070137 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-ca-bundle\") pod \"16519122-bb38-4aa4-980b-82b7edd5cdc5\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " Oct 30 18:42:22.070446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070169 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-conf\") pod \"16519122-bb38-4aa4-980b-82b7edd5cdc5\" (UID: \"16519122-bb38-4aa4-980b-82b7edd5cdc5\") " Oct 30 18:42:22.070446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070217 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-conf\") pod \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " Oct 30 18:42:22.070446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070249 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-kube-api-access-774zf\" (UniqueName: \"kubernetes.io/projected/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-kube-api-access-774zf\") pod \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " Oct 30 18:42:22.070446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070284 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-ca-bundle\") pod \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\" (UID: \"8d1ff3f9-58e9-472b-bde3-f17be51ea7de\") " Oct 30 18:42:22.070701 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070465 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-tmp-storage" pod "8d1ff3f9-58e9-472b-bde3-f17be51ea7de" (UID: "8d1ff3f9-58e9-472b-bde3-f17be51ea7de"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:22.070701 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070475 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-tmp-storage" pod "16519122-bb38-4aa4-980b-82b7edd5cdc5" (UID: "16519122-bb38-4aa4-980b-82b7edd5cdc5"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:22.070701 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070578 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-tmp-storage\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:22.070701 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070626 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-tmp-storage\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:22.070701 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070634 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-ca-bundle" pod "16519122-bb38-4aa4-980b-82b7edd5cdc5" (UID: "16519122-bb38-4aa4-980b-82b7edd5cdc5"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:22.070939 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070868 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-ca-bundle" pod "8d1ff3f9-58e9-472b-bde3-f17be51ea7de" (UID: "8d1ff3f9-58e9-472b-bde3-f17be51ea7de"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:22.070939 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.070878 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-conf" pod "16519122-bb38-4aa4-980b-82b7edd5cdc5" (UID: "16519122-bb38-4aa4-980b-82b7edd5cdc5"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:22.071179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.071156 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-conf" pod "8d1ff3f9-58e9-472b-bde3-f17be51ea7de" (UID: "8d1ff3f9-58e9-472b-bde3-f17be51ea7de"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:22.072477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.072451 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16519122-bb38-4aa4-980b-82b7edd5cdc5-kube-api-access-fgmbz" pod "16519122-bb38-4aa4-980b-82b7edd5cdc5" (UID: "16519122-bb38-4aa4-980b-82b7edd5cdc5"). InnerVolumeSpecName "kube-api-access-fgmbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:22.072571 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.072478 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-querier-mtls" pod "16519122-bb38-4aa4-980b-82b7edd5cdc5" (UID: "16519122-bb38-4aa4-980b-82b7edd5cdc5"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:22.072571 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.072523 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-kube-api-access-774zf" pod "8d1ff3f9-58e9-472b-bde3-f17be51ea7de" (UID: "8d1ff3f9-58e9-472b-bde3-f17be51ea7de"). InnerVolumeSpecName "kube-api-access-774zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:22.072792 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.072771 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-query-frontend-mtls" pod "8d1ff3f9-58e9-472b-bde3-f17be51ea7de" (UID: "8d1ff3f9-58e9-472b-bde3-f17be51ea7de"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:22.164685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.164658 5014 generic.go:356] "Generic (PLEG): container finished" podID="8d1ff3f9-58e9-472b-bde3-f17be51ea7de" containerID="ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393" exitCode=137 Oct 30 18:42:22.164803 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.164699 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" Oct 30 18:42:22.164803 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.164750 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" event={"ID":"8d1ff3f9-58e9-472b-bde3-f17be51ea7de","Type":"ContainerDied","Data":"ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393"} Oct 30 18:42:22.164924 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.164802 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk" event={"ID":"8d1ff3f9-58e9-472b-bde3-f17be51ea7de","Type":"ContainerDied","Data":"78840f38f66667e609e163d12c84cfcf0407fb7461a66f7ca766a63ce7f1a81e"} Oct 30 18:42:22.164924 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.164833 5014 scope.go:122] "RemoveContainer" containerID="ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393" Oct 30 18:42:22.166121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.166094 5014 generic.go:356] "Generic (PLEG): container finished" podID="16519122-bb38-4aa4-980b-82b7edd5cdc5" containerID="84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e" exitCode=137 Oct 30 18:42:22.166213 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.166126 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" Oct 30 18:42:22.166213 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.166169 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" event={"ID":"16519122-bb38-4aa4-980b-82b7edd5cdc5","Type":"ContainerDied","Data":"84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e"} Oct 30 18:42:22.166317 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.166217 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc" event={"ID":"16519122-bb38-4aa4-980b-82b7edd5cdc5","Type":"ContainerDied","Data":"34a9be4779ba0da4aa91f606477336850677128a8c32cf84870a718752491a1e"} Oct 30 18:42:22.171622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.171570 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:22.171717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.171628 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:22.171717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.171645 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:22.171717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.171659 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-774zf\" (UniqueName: \"kubernetes.io/projected/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-kube-api-access-774zf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:22.171717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.171673 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:22.171717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.171688 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/8d1ff3f9-58e9-472b-bde3-f17be51ea7de-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:22.171717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.171702 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgmbz\" (UniqueName: \"kubernetes.io/projected/16519122-bb38-4aa4-980b-82b7edd5cdc5-kube-api-access-fgmbz\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:22.171717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.171716 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/16519122-bb38-4aa4-980b-82b7edd5cdc5-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:22.174114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.174069 5014 scope.go:122] "RemoveContainer" containerID="ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393" Oct 30 18:42:22.174425 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:42:22.174406 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393\": container with ID starting with ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393 not found: ID does not exist" containerID="ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393" Oct 30 18:42:22.174500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.174431 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393"} err="failed to get container status \"ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393\": rpc error: code = NotFound desc = could not find container \"ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393\": container with ID starting with ba44d3fcd5e331a98ba790dae7b47f01f6494f0655107e9a778b98bf9f094393 not found: ID does not exist" Oct 30 18:42:22.174500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.174457 5014 scope.go:122] "RemoveContainer" containerID="84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e" Oct 30 18:42:22.182659 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.182636 5014 scope.go:122] "RemoveContainer" containerID="84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e" Oct 30 18:42:22.182916 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:42:22.182897 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e\": container with ID starting with 84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e not found: ID does not exist" containerID="84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e" Oct 30 18:42:22.182987 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.182925 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e"} err="failed to get container status \"84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e\": rpc error: code = NotFound desc = could not find container \"84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e\": container with ID starting with 84c32d6af350ef3863745532bc8b0e5fe78f0fb96333111f9224c5aebbda3a9e not found: ID does not exist" Oct 30 18:42:22.215016 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.214992 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk"] Oct 30 18:42:22.224135 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.224113 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7f6b8d55-42rvk"] Oct 30 18:42:22.258065 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.258041 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc"] Oct 30 18:42:22.266053 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:22.266033 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5958499c6b-dvvvc"] Oct 30 18:42:23.141681 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:23.141651 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16519122-bb38-4aa4-980b-82b7edd5cdc5" path="/var/lib/kubelet/pods/16519122-bb38-4aa4-980b-82b7edd5cdc5/volumes" Oct 30 18:42:23.142066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:23.142033 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1ff3f9-58e9-472b-bde3-f17be51ea7de" path="/var/lib/kubelet/pods/8d1ff3f9-58e9-472b-bde3-f17be51ea7de/volumes" Oct 30 18:42:32.385680 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.385643 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"] Oct 30 18:42:32.386340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386318 5014 cpu_manager.go:414] "containerMap: removing container" podUID="2717e179-c335-42d2-864a-9759944dc93a" containerName="tempo" Oct 30 18:42:32.386445 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386342 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2717e179-c335-42d2-864a-9759944dc93a" containerName="tempo" Oct 30 18:42:32.386445 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386369 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b522fc71-c6ce-4aa6-81f7-77b8534502e5" containerName="tempo" Oct 30 18:42:32.386445 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386377 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b522fc71-c6ce-4aa6-81f7-77b8534502e5" containerName="tempo" Oct 30 18:42:32.386445 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386394 5014 cpu_manager.go:414] "containerMap: removing container" podUID="16519122-bb38-4aa4-980b-82b7edd5cdc5" containerName="tempo" Oct 30 18:42:32.386445 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386402 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16519122-bb38-4aa4-980b-82b7edd5cdc5" containerName="tempo" Oct 30 18:42:32.386445 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386432 5014 cpu_manager.go:414] "containerMap: removing container" podUID="8136e1c7-0e30-4fc0-a878-1a7402a2c130" containerName="tempo" Oct 30 18:42:32.386445 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386439 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8136e1c7-0e30-4fc0-a878-1a7402a2c130" containerName="tempo" Oct 30 18:42:32.386815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386457 5014 cpu_manager.go:414] "containerMap: removing container" podUID="394e9329-47b8-4454-a1d0-c79f386b5cf1" containerName="otc-container" Oct 30 18:42:32.386815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386465 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394e9329-47b8-4454-a1d0-c79f386b5cf1" containerName="otc-container" Oct 30 18:42:32.386815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386519 5014 cpu_manager.go:414] "containerMap: removing container" podUID="8d1ff3f9-58e9-472b-bde3-f17be51ea7de" containerName="tempo" Oct 30 18:42:32.386815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386527 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1ff3f9-58e9-472b-bde3-f17be51ea7de" containerName="tempo" Oct 30 18:42:32.386815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386734 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="2717e179-c335-42d2-864a-9759944dc93a" containerName="tempo" Oct 30 18:42:32.386815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386752 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b522fc71-c6ce-4aa6-81f7-77b8534502e5" containerName="tempo" Oct 30 18:42:32.386815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386762 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="16519122-bb38-4aa4-980b-82b7edd5cdc5" containerName="tempo" Oct 30 18:42:32.386815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386782 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="8136e1c7-0e30-4fc0-a878-1a7402a2c130" containerName="tempo" Oct 30 18:42:32.386815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386798 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="394e9329-47b8-4454-a1d0-c79f386b5cf1" containerName="otc-container" Oct 30 18:42:32.386815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.386811 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1ff3f9-58e9-472b-bde3-f17be51ea7de" containerName="tempo" Oct 30 18:42:32.391785 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.391763 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 30 18:42:32.406540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.406511 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"] Oct 30 18:42:32.406768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.406748 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:42:32.416892 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.416873 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:42:32.417839 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.417821 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-5mtd6\"" Oct 30 18:42:32.544764 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.544741 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s5nm\" (UniqueName: \"kubernetes.io/projected/caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd-kube-api-access-7s5nm\") pod \"fake-gcs-bucket-creator\" (UID: \"caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd\") " pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 30 18:42:32.645654 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.645562 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s5nm\" (UniqueName: \"kubernetes.io/projected/caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd-kube-api-access-7s5nm\") pod \"fake-gcs-bucket-creator\" (UID: \"caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd\") " pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 30 18:42:32.669003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.668980 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s5nm\" (UniqueName: \"kubernetes.io/projected/caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd-kube-api-access-7s5nm\") pod \"fake-gcs-bucket-creator\" (UID: \"caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd\") " pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 30 18:42:32.710265 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:32.710241 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 30 18:42:33.004952 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:33.004883 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"] Oct 30 18:42:33.009280 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:42:33.009249 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaa96ec4_aabd_4ba6_8aa9_ace3b533e4dd.slice/crio-31193c0d6f3f6e97a42f014ded40698f90287d7e28ab2178802a1f73ae2f6935 WatchSource:0}: Error finding container 31193c0d6f3f6e97a42f014ded40698f90287d7e28ab2178802a1f73ae2f6935: Status 404 returned error can't find the container with id 31193c0d6f3f6e97a42f014ded40698f90287d7e28ab2178802a1f73ae2f6935 Oct 30 18:42:33.214959 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:33.214926 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" event={"ID":"caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd","Type":"ContainerStarted","Data":"31193c0d6f3f6e97a42f014ded40698f90287d7e28ab2178802a1f73ae2f6935"} Oct 30 18:42:34.220403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:34.220371 5014 generic.go:356] "Generic (PLEG): container finished" podID="caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd" containerID="376d9b1c7312a2978eda4425bff513dd6695f8f0fd2efec01d4fcffcf49796e8" exitCode=0 Oct 30 18:42:34.220747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:34.220450 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" event={"ID":"caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd","Type":"ContainerDied","Data":"376d9b1c7312a2978eda4425bff513dd6695f8f0fd2efec01d4fcffcf49796e8"} Oct 30 18:42:35.359315 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:35.359294 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 30 18:42:35.468296 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:35.468268 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd-kube-api-access-7s5nm\" (UniqueName: \"kubernetes.io/projected/caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd-kube-api-access-7s5nm\") pod \"caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd\" (UID: \"caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd\") " Oct 30 18:42:35.470351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:35.470328 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd-kube-api-access-7s5nm" pod "caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd" (UID: "caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd"). InnerVolumeSpecName "kube-api-access-7s5nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:35.568878 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:35.568852 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s5nm\" (UniqueName: \"kubernetes.io/projected/caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd-kube-api-access-7s5nm\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:36.229096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:36.229059 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" event={"ID":"caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd","Type":"ContainerDied","Data":"31193c0d6f3f6e97a42f014ded40698f90287d7e28ab2178802a1f73ae2f6935"} Oct 30 18:42:36.229096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:36.229087 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 30 18:42:36.229289 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:36.229102 5014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31193c0d6f3f6e97a42f014ded40698f90287d7e28ab2178802a1f73ae2f6935" Oct 30 18:42:38.220680 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.220637 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:42:38.221136 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.221121 5014 cpu_manager.go:414] "containerMap: removing container" podUID="caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd" containerName="create-bucket" Oct 30 18:42:38.221188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.221139 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd" containerName="create-bucket" Oct 30 18:42:38.221266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.221255 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd" containerName="create-bucket" Oct 30 18:42:38.224642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.224621 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.249426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.248506 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:42:38.249582 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.248815 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:42:38.251014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.250991 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:42:38.251148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.251134 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-qcbgc\"" Oct 30 18:42:38.251228 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.251217 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:42:38.253374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.253353 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:42:38.260264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.260246 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 18:42:38.387464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.387427 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndj5\" (UniqueName: \"kubernetes.io/projected/461bf434-3215-4c93-9095-adc018d78854-kube-api-access-zndj5\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.387671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.387490 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.387671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.387611 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/461bf434-3215-4c93-9095-adc018d78854-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.387671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.387644 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.488345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.488266 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.488345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.488307 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/461bf434-3215-4c93-9095-adc018d78854-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.488345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.488323 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.488553 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.488365 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndj5\" (UniqueName: \"kubernetes.io/projected/461bf434-3215-4c93-9095-adc018d78854-kube-api-access-zndj5\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.489106 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.489077 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/461bf434-3215-4c93-9095-adc018d78854-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.490887 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.490860 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.491121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.491099 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.511495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.511470 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndj5\" (UniqueName: \"kubernetes.io/projected/461bf434-3215-4c93-9095-adc018d78854-kube-api-access-zndj5\") pod \"data-science-collector-collector-0\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.539127 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.539108 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:42:38.839668 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.839635 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:42:38.844763 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:42:38.844725 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461bf434_3215_4c93_9095_adc018d78854.slice/crio-a3be19b9634b9a97e9c187d511521c9d61fb64b633b054e03db25ad33a9f8bd8 WatchSource:0}: Error finding container a3be19b9634b9a97e9c187d511521c9d61fb64b633b054e03db25ad33a9f8bd8: Status 404 returned error can't find the container with id a3be19b9634b9a97e9c187d511521c9d61fb64b633b054e03db25ad33a9f8bd8 Oct 30 18:42:38.847093 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.847063 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:42:38.847191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:38.847124 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:42:39.246511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:39.246424 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"461bf434-3215-4c93-9095-adc018d78854","Type":"ContainerStarted","Data":"7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20"} Oct 30 18:42:39.246511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:39.246478 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"461bf434-3215-4c93-9095-adc018d78854","Type":"ContainerStarted","Data":"a3be19b9634b9a97e9c187d511521c9d61fb64b633b054e03db25ad33a9f8bd8"} Oct 30 18:42:39.288631 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:39.288570 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.288556791 podStartE2EDuration="1.288556791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:39.287291497 +0000 UTC m=+2772.275631995" watchObservedRunningTime="2026-10-30 18:42:39.288556791 +0000 UTC m=+2772.276897278" Oct 30 18:42:39.971335 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:39.971305 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85"] Oct 30 18:42:39.975204 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:39.975184 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.000877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.000850 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-vf8g6\"" Oct 30 18:42:40.001008 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.000977 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\"" Oct 30 18:42:40.001081 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.001035 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 30 18:42:40.001322 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.001293 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\"" Oct 30 18:42:40.002979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.002960 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85"] Oct 30 18:42:40.040704 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.040678 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:42:40.044408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.044359 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.066804 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.066780 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:42:40.089991 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.089965 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"gcs-secret\"" Oct 30 18:42:40.090125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.090107 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\"" Oct 30 18:42:40.102127 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.102094 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w899x\" (UniqueName: \"kubernetes.io/projected/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-kube-api-access-w899x\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.102248 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.102137 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.102248 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.102215 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.102335 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.102253 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.102400 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.102378 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.179008 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.178976 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6"] Oct 30 18:42:40.182523 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.182503 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.197937 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.197919 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\"" Oct 30 18:42:40.203292 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.203266 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgbbd\" (UniqueName: \"kubernetes.io/projected/390ad5ee-1d36-4f30-9e02-679eba7df0b3-kube-api-access-cgbbd\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.203392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.203363 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.203500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.203415 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.203500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.203473 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.203622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.203506 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.203622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.203535 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.203622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.203576 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.203826 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.203649 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w899x\" (UniqueName: \"kubernetes.io/projected/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-kube-api-access-w899x\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.203826 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.203686 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.203826 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.203720 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.204723 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.204266 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.204841 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.204756 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.205338 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.205317 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6"] Oct 30 18:42:40.206147 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.205624 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.208053 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.208030 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.227962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.227892 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp"] Oct 30 18:42:40.231971 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.231953 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.234877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.234857 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w899x\" (UniqueName: \"kubernetes.io/projected/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-kube-api-access-w899x\") pod \"tempo-data-science-tempostack-distributor-9644b84bf-mnq85\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.252174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.252078 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp"] Oct 30 18:42:40.288477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.288455 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:40.298150 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.298126 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\"" Oct 30 18:42:40.301098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.301067 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5"] Oct 30 18:42:40.304137 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304116 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.304247 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304145 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.304247 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304175 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.304247 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304203 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l77b4\" (UniqueName: \"kubernetes.io/projected/c00c4728-ce69-493a-859f-8aa4eb2161ec-kube-api-access-l77b4\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.304435 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304376 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.304435 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304427 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgbbd\" (UniqueName: \"kubernetes.io/projected/390ad5ee-1d36-4f30-9e02-679eba7df0b3-kube-api-access-cgbbd\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.304545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304457 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-conf\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.304545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304514 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.304545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304540 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.304715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304567 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.304715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304617 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.304715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304654 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.304715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304688 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.304917 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304718 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt85c\" (UniqueName: \"kubernetes.io/projected/f2129aca-0d65-412e-bc81-f90e40f10eb3-kube-api-access-tt85c\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.304917 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304745 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.304917 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304768 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.304917 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.304778 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.305466 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.305291 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.306125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.306085 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.306125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.306121 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.307753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.307730 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.308412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.308391 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.326475 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.326445 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5"] Oct 30 18:42:40.326956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.326936 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\"" Oct 30 18:42:40.328842 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.328817 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgbbd\" (UniqueName: \"kubernetes.io/projected/390ad5ee-1d36-4f30-9e02-679eba7df0b3-kube-api-access-cgbbd\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:40.406538 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406508 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.406695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406553 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.406695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406580 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.406835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406722 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt85c\" (UniqueName: \"kubernetes.io/projected/f2129aca-0d65-412e-bc81-f90e40f10eb3-kube-api-access-tt85c\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.406835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406751 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.406835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406770 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.406835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406788 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.407040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406848 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.407040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406870 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.407040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406895 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.407040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406926 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.407040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.406958 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l77b4\" (UniqueName: \"kubernetes.io/projected/c00c4728-ce69-493a-859f-8aa4eb2161ec-kube-api-access-l77b4\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.407040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.407018 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kphzr\" (UniqueName: \"kubernetes.io/projected/27f44482-f8d8-4e8c-9dab-6fae485ca581-kube-api-access-kphzr\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.407320 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.407059 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.407320 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.407101 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-conf\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.407320 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.407127 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.407320 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.407136 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.407320 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.407208 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.407320 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.407246 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.407654 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.407524 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.407654 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.407542 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.408216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.408190 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.408549 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.408520 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.408946 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.408925 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-conf\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.409784 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.409758 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.409881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.409758 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.410390 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.410368 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.410857 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.410836 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.428861 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.428843 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt85c\" (UniqueName: \"kubernetes.io/projected/f2129aca-0d65-412e-bc81-f90e40f10eb3-kube-api-access-tt85c\") pod \"tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.429976 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.429953 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l77b4\" (UniqueName: \"kubernetes.io/projected/c00c4728-ce69-493a-859f-8aa4eb2161ec-kube-api-access-l77b4\") pod \"tempo-data-science-tempostack-querier-687b9b5c79-87cvp\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.494984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.494940 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:40.508177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.508157 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kphzr\" (UniqueName: \"kubernetes.io/projected/27f44482-f8d8-4e8c-9dab-6fae485ca581-kube-api-access-kphzr\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.508265 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.508201 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.508265 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.508239 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.508390 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.508266 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.508390 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.508305 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.508390 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.508354 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.508957 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.508915 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.509052 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.509014 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.509473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.509450 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.510998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.510921 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.511692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.511663 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.532856 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.532804 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kphzr\" (UniqueName: \"kubernetes.io/projected/27f44482-f8d8-4e8c-9dab-6fae485ca581-kube-api-access-kphzr\") pod \"tempo-data-science-tempostack-compactor-854c845df5-cmhx5\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.554948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.554922 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:40.613805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.613547 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85"] Oct 30 18:42:40.616068 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:42:40.616030 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dd342e_5dc7_4057_ac59_9ea1f8e244b7.slice/crio-0b9f8c84c6f887dadbb8c097bf8aed0a66c09beada622bbf5763b95b9992ac56 WatchSource:0}: Error finding container 0b9f8c84c6f887dadbb8c097bf8aed0a66c09beada622bbf5763b95b9992ac56: Status 404 returned error can't find the container with id 0b9f8c84c6f887dadbb8c097bf8aed0a66c09beada622bbf5763b95b9992ac56 Oct 30 18:42:40.626527 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.626023 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:40.650898 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.650875 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7"] Oct 30 18:42:40.657758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.656850 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.673892 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.673265 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 18:42:40.673892 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.673287 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 30 18:42:40.673892 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.673413 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 30 18:42:40.673892 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.673848 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-vwtls\"" Oct 30 18:42:40.674218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.674105 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 18:42:40.674218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.674187 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 30 18:42:40.676268 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.676244 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7"] Oct 30 18:42:40.714174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.714144 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.714276 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.714204 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-serving-certs\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.714276 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.714256 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5hf\" (UniqueName: \"kubernetes.io/projected/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-kube-api-access-ch5hf\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.714349 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.714291 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tenant\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.714398 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.714346 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-rbac\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.714398 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.714372 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.714507 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.714398 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-cabundle\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.797662 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.797558 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6"] Oct 30 18:42:40.802521 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:42:40.802494 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2129aca_0d65_412e_bc81_f90e40f10eb3.slice/crio-e26b896fffb3e4fb5b188aa754d1ae628c068253e467e9a399ee6d4081eb73b5 WatchSource:0}: Error finding container e26b896fffb3e4fb5b188aa754d1ae628c068253e467e9a399ee6d4081eb73b5: Status 404 returned error can't find the container with id e26b896fffb3e4fb5b188aa754d1ae628c068253e467e9a399ee6d4081eb73b5 Oct 30 18:42:40.814897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.814869 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-rbac\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.815018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.814903 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.815018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.814919 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-cabundle\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.815018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.814988 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.815191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.815026 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-serving-certs\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.815191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.815078 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5hf\" (UniqueName: \"kubernetes.io/projected/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-kube-api-access-ch5hf\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.815191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.815113 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tenant\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.815847 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.815822 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-rbac\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.816072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.816028 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.817340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.817296 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-cabundle\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.819518 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.819472 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.819518 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.819488 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tenant\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.820040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.820020 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-serving-certs\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.840714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.840688 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5hf\" (UniqueName: \"kubernetes.io/projected/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-kube-api-access-ch5hf\") pod \"tempo-data-science-tempostack-gateway-b65b4f79f-h28k7\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:40.858418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.858395 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp"] Oct 30 18:42:40.862394 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:42:40.862369 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00c4728_ce69_493a_859f_8aa4eb2161ec.slice/crio-06fd4228f9c019d8b08debd58b85354736f7ed9b24ba0f184460a2944208b00f WatchSource:0}: Error finding container 06fd4228f9c019d8b08debd58b85354736f7ed9b24ba0f184460a2944208b00f: Status 404 returned error can't find the container with id 06fd4228f9c019d8b08debd58b85354736f7ed9b24ba0f184460a2944208b00f Oct 30 18:42:40.960947 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.960913 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5"] Oct 30 18:42:40.965399 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:42:40.965367 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f44482_f8d8_4e8c_9dab_6fae485ca581.slice/crio-349e41aec40a06a500cfe6c193a31725ddac53c9dc48519b54f92601eca3baa7 WatchSource:0}: Error finding container 349e41aec40a06a500cfe6c193a31725ddac53c9dc48519b54f92601eca3baa7: Status 404 returned error can't find the container with id 349e41aec40a06a500cfe6c193a31725ddac53c9dc48519b54f92601eca3baa7 Oct 30 18:42:40.968621 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.968569 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:42:40.986331 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:42:40.986270 5014 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[data], unattached volumes=[data], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="390ad5ee-1d36-4f30-9e02-679eba7df0b3" Oct 30 18:42:40.998272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:40.998243 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:41.257732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.257695 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" event={"ID":"f2129aca-0d65-412e-bc81-f90e40f10eb3","Type":"ContainerStarted","Data":"47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411"} Oct 30 18:42:41.258155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.257761 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" event={"ID":"f2129aca-0d65-412e-bc81-f90e40f10eb3","Type":"ContainerStarted","Data":"e26b896fffb3e4fb5b188aa754d1ae628c068253e467e9a399ee6d4081eb73b5"} Oct 30 18:42:41.258155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.257895 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:42:41.259625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.259573 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" event={"ID":"c00c4728-ce69-493a-859f-8aa4eb2161ec","Type":"ContainerStarted","Data":"1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb"} Oct 30 18:42:41.259750 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.259644 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:42:41.259750 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.259668 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" event={"ID":"c00c4728-ce69-493a-859f-8aa4eb2161ec","Type":"ContainerStarted","Data":"06fd4228f9c019d8b08debd58b85354736f7ed9b24ba0f184460a2944208b00f"} Oct 30 18:42:41.261222 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.261193 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" event={"ID":"40dd342e-5dc7-4057-ac59-9ea1f8e244b7","Type":"ContainerStarted","Data":"cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352"} Oct 30 18:42:41.261315 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.261239 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" event={"ID":"40dd342e-5dc7-4057-ac59-9ea1f8e244b7","Type":"ContainerStarted","Data":"0b9f8c84c6f887dadbb8c097bf8aed0a66c09beada622bbf5763b95b9992ac56"} Oct 30 18:42:41.261360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.261311 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:42:41.262885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.262867 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:41.262990 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.262868 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" event={"ID":"27f44482-f8d8-4e8c-9dab-6fae485ca581","Type":"ContainerStarted","Data":"ab293791839df0ded9ef871fc667ace32751077e3ce8b2936a77ef0440551c5a"} Oct 30 18:42:41.263034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.263015 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" event={"ID":"27f44482-f8d8-4e8c-9dab-6fae485ca581","Type":"ContainerStarted","Data":"349e41aec40a06a500cfe6c193a31725ddac53c9dc48519b54f92601eca3baa7"} Oct 30 18:42:41.263071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.263039 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:42:41.269572 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.269558 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:41.299384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.299344 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" podStartSLOduration=1.299332108 podStartE2EDuration="1.299332108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:41.29778864 +0000 UTC m=+2774.286129128" watchObservedRunningTime="2026-10-30 18:42:41.299332108 +0000 UTC m=+2774.287672594" Oct 30 18:42:41.317026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.316971 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7"] Oct 30 18:42:41.319772 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.319747 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390ad5ee-1d36-4f30-9e02-679eba7df0b3-kube-api-access-cgbbd\" (UniqueName: \"kubernetes.io/projected/390ad5ee-1d36-4f30-9e02-679eba7df0b3-kube-api-access-cgbbd\") pod \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " Oct 30 18:42:41.319831 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.319789 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-storage-gcs-key\") pod \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " Oct 30 18:42:41.319864 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.319828 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ingester-mtls\") pod \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " Oct 30 18:42:41.319895 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.319873 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ca-bundle\") pod \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " Oct 30 18:42:41.319942 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.319926 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-conf\") pod \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\" (UID: \"390ad5ee-1d36-4f30-9e02-679eba7df0b3\") " Oct 30 18:42:41.322214 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.320815 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-conf" pod "390ad5ee-1d36-4f30-9e02-679eba7df0b3" (UID: "390ad5ee-1d36-4f30-9e02-679eba7df0b3"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:41.322368 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:42:41.322348 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf935b6e1_fc16_4cce_ad16_2cd7bc8cdb72.slice/crio-64dcf33f8456a341f85624fdebbebe7dc8c462132784839d2177e68242f0415c WatchSource:0}: Error finding container 64dcf33f8456a341f85624fdebbebe7dc8c462132784839d2177e68242f0415c: Status 404 returned error can't find the container with id 64dcf33f8456a341f85624fdebbebe7dc8c462132784839d2177e68242f0415c Oct 30 18:42:41.322655 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.322624 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ingester-mtls" pod "390ad5ee-1d36-4f30-9e02-679eba7df0b3" (UID: "390ad5ee-1d36-4f30-9e02-679eba7df0b3"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:41.322785 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.322758 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-storage-gcs-key" pod "390ad5ee-1d36-4f30-9e02-679eba7df0b3" (UID: "390ad5ee-1d36-4f30-9e02-679eba7df0b3"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:41.322899 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.322785 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390ad5ee-1d36-4f30-9e02-679eba7df0b3-kube-api-access-cgbbd" pod "390ad5ee-1d36-4f30-9e02-679eba7df0b3" (UID: "390ad5ee-1d36-4f30-9e02-679eba7df0b3"). InnerVolumeSpecName "kube-api-access-cgbbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:41.322899 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.322801 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ca-bundle" pod "390ad5ee-1d36-4f30-9e02-679eba7df0b3" (UID: "390ad5ee-1d36-4f30-9e02-679eba7df0b3"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:41.339446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.339412 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" podStartSLOduration=2.33940041 podStartE2EDuration="2.33940041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:41.338495162 +0000 UTC m=+2774.326835678" watchObservedRunningTime="2026-10-30 18:42:41.33940041 +0000 UTC m=+2774.327740898" Oct 30 18:42:41.419836 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.419785 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" podStartSLOduration=1.419770231 podStartE2EDuration="1.419770231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:41.419322033 +0000 UTC m=+2774.407662520" watchObservedRunningTime="2026-10-30 18:42:41.419770231 +0000 UTC m=+2774.408110721" Oct 30 18:42:41.421351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.421330 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgbbd\" (UniqueName: \"kubernetes.io/projected/390ad5ee-1d36-4f30-9e02-679eba7df0b3-kube-api-access-cgbbd\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.421431 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.421353 5014 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-storage-gcs-key\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.421431 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.421363 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.421431 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.421373 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.421431 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.421381 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/390ad5ee-1d36-4f30-9e02-679eba7df0b3-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.464326 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:41.464285 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" podStartSLOduration=1.464271868 podStartE2EDuration="1.464271868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:41.46354443 +0000 UTC m=+2774.451884918" watchObservedRunningTime="2026-10-30 18:42:41.464271868 +0000 UTC m=+2774.452612356" Oct 30 18:42:42.272524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.271854 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" event={"ID":"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72","Type":"ContainerStarted","Data":"fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00"} Oct 30 18:42:42.272524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.271922 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" event={"ID":"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72","Type":"ContainerStarted","Data":"f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e"} Oct 30 18:42:42.272524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.271942 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" event={"ID":"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72","Type":"ContainerStarted","Data":"64dcf33f8456a341f85624fdebbebe7dc8c462132784839d2177e68242f0415c"} Oct 30 18:42:42.272524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.272138 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.331799 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.331755 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" podStartSLOduration=2.331742075 podStartE2EDuration="2.331742075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:42.330908778 +0000 UTC m=+2775.319249265" watchObservedRunningTime="2026-10-30 18:42:42.331742075 +0000 UTC m=+2775.320082615" Oct 30 18:42:42.392129 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.392096 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:42:42.397755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.397733 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:42:42.436392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.436326 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:42:42.440303 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.440285 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.455867 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.455848 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\"" Oct 30 18:42:42.476739 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.476718 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:42:42.532776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.532754 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.532905 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.532790 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4s8q\" (UniqueName: \"kubernetes.io/projected/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-kube-api-access-b4s8q\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.532905 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.532873 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.532905 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.532900 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.533032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.532929 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.633585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.633548 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4s8q\" (UniqueName: \"kubernetes.io/projected/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-kube-api-access-b4s8q\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.633761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.633678 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.633761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.633715 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.633761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.633743 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.633914 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.633796 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.634447 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.634424 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.634557 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.634519 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.636267 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.636245 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.636357 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.636291 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:42.656697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:42.656678 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4s8q\" (UniqueName: \"kubernetes.io/projected/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-kube-api-access-b4s8q\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:43.142495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:43.142464 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390ad5ee-1d36-4f30-9e02-679eba7df0b3" path="/var/lib/kubelet/pods/390ad5ee-1d36-4f30-9e02-679eba7df0b3/volumes" Oct 30 18:42:43.276448 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:43.276419 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:43.276871 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:43.276480 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:43.277781 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:43.277751 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" podUID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.102:8083/ready\": dial tcp 10.132.0.102:8083: connect: connection refused" Oct 30 18:42:43.332675 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:42:43.332645 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:42:43.482046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:43.481975 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:44.285339 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:44.285306 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:44.313827 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:44.313801 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7"] Oct 30 18:42:45.154474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:45.154440 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:45.183375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:45.183346 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:45.255209 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:45.255183 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:45.255318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:45.255278 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:45.269400 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:45.269378 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") DevicePath \"csi-2de5aa6ecd623838765e0f31befbb65b2679c3ca7e322e1ba1051ac031e970ac\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:45.372760 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:45.372734 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a4e73d0eb00dee7b83c74b37dc76bd22c1b434e850c00b254cd96b669149949b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:45.399721 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:45.399695 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:45.455422 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:45.455369 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:45.767973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:45.767896 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:42:45.772172 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:42:45.772146 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be88b2b_0140_4e66_bfc6_2ab58e64e57d.slice/crio-a0f6e91df190332fbb2f1661167c7cb77243fbe6a2052858901b0a134558aee6 WatchSource:0}: Error finding container a0f6e91df190332fbb2f1661167c7cb77243fbe6a2052858901b0a134558aee6: Status 404 returned error can't find the container with id a0f6e91df190332fbb2f1661167c7cb77243fbe6a2052858901b0a134558aee6 Oct 30 18:42:46.294501 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:46.294464 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"4be88b2b-0140-4e66-bfc6-2ab58e64e57d","Type":"ContainerStarted","Data":"1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798"} Oct 30 18:42:46.294733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:46.294525 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"4be88b2b-0140-4e66-bfc6-2ab58e64e57d","Type":"ContainerStarted","Data":"a0f6e91df190332fbb2f1661167c7cb77243fbe6a2052858901b0a134558aee6"} Oct 30 18:42:46.294733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:46.294542 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" podUID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" containerName="tempo-gateway" containerID="cri-o://f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e" gracePeriod=30 Oct 30 18:42:46.294733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:46.294556 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:42:46.294733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:46.294586 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" podUID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" containerName="tempo-gateway-opa" containerID="cri-o://fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00" gracePeriod=30 Oct 30 18:42:46.393851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:46.393812 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podStartSLOduration=4.393797452 podStartE2EDuration="4.393797452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:46.393179968 +0000 UTC m=+2779.381520455" watchObservedRunningTime="2026-10-30 18:42:46.393797452 +0000 UTC m=+2779.382137939" Oct 30 18:42:47.300976 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.300941 5014 generic.go:356] "Generic (PLEG): container finished" podID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" containerID="fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00" exitCode=0 Oct 30 18:42:47.301119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.301006 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" event={"ID":"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72","Type":"ContainerDied","Data":"fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00"} Oct 30 18:42:47.553319 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.553294 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:47.676816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.676787 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-gateway-mtls\") pod \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " Oct 30 18:42:47.676982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.676861 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-kube-api-access-ch5hf\" (UniqueName: \"kubernetes.io/projected/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-kube-api-access-ch5hf\") pod \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " Oct 30 18:42:47.676982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.676893 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-rbac\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-rbac\") pod \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " Oct 30 18:42:47.677098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.676982 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tenant\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tenant\") pod \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " Oct 30 18:42:47.677098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.677011 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-serving-certs\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-serving-certs\") pod \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " Oct 30 18:42:47.677098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.677057 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-cabundle\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-cabundle\") pod \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " Oct 30 18:42:47.677098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.677092 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-ca-bundle\") pod \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\" (UID: \"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72\") " Oct 30 18:42:47.677403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.677368 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-rbac" pod "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" (UID: "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:47.677491 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.677414 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-cabundle" pod "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" (UID: "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:47.677696 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.677677 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-ca-bundle" pod "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" (UID: "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:47.679163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.679145 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-serving-certs" pod "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" (UID: "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:47.679568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.679542 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-kube-api-access-ch5hf" pod "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" (UID: "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72"). InnerVolumeSpecName "kube-api-access-ch5hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:47.679568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.679548 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-gateway-mtls" pod "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" (UID: "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:47.685260 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.685236 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tenant" pod "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" (UID: "f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:47.778000 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.777979 5014 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tenant\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:47.778000 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.777999 5014 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-serving-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:47.778117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.778009 5014 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-cabundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:47.778117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.778017 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:47.778117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.778026 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:47.778117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.778036 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch5hf\" (UniqueName: \"kubernetes.io/projected/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-kube-api-access-ch5hf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:47.778117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:47.778045 5014 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:42:48.306801 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.306766 5014 generic.go:356] "Generic (PLEG): container finished" podID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" containerID="f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e" exitCode=0 Oct 30 18:42:48.306984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.306823 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" Oct 30 18:42:48.306984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.306854 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" event={"ID":"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72","Type":"ContainerDied","Data":"f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e"} Oct 30 18:42:48.306984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.306921 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7" event={"ID":"f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72","Type":"ContainerDied","Data":"64dcf33f8456a341f85624fdebbebe7dc8c462132784839d2177e68242f0415c"} Oct 30 18:42:48.306984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.306959 5014 scope.go:122] "RemoveContainer" containerID="fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00" Oct 30 18:42:48.316198 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.316178 5014 scope.go:122] "RemoveContainer" containerID="f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e" Oct 30 18:42:48.324484 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.324468 5014 scope.go:122] "RemoveContainer" containerID="fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00" Oct 30 18:42:48.324747 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:42:48.324728 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00\": container with ID starting with fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00 not found: ID does not exist" containerID="fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00" Oct 30 18:42:48.324830 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.324757 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00"} err="failed to get container status \"fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00\": rpc error: code = NotFound desc = could not find container \"fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00\": container with ID starting with fa4edd575efe4cabcb415f69f3cbe268748b940d3cb7cc9f063b3ab9b86f8c00 not found: ID does not exist" Oct 30 18:42:48.324830 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.324782 5014 scope.go:122] "RemoveContainer" containerID="f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e" Oct 30 18:42:48.325021 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:42:48.325005 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e\": container with ID starting with f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e not found: ID does not exist" containerID="f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e" Oct 30 18:42:48.325065 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.325028 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e"} err="failed to get container status \"f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e\": rpc error: code = NotFound desc = could not find container \"f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e\": container with ID starting with f98e4a7f075a730ca71a5a6cd006633408ee7b179dd2913f575bad17b32ba47e not found: ID does not exist" Oct 30 18:42:48.370116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.370095 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7"] Oct 30 18:42:48.382006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:48.381983 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b65b4f79f-h28k7"] Oct 30 18:42:49.141488 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:42:49.141454 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" path="/var/lib/kubelet/pods/f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72/volumes" Oct 30 18:43:02.278765 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:02.278730 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:43:02.279315 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:02.279235 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:43:02.279396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:02.279371 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" podUID="27f44482-f8d8-4e8c-9dab-6fae485ca581" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:02.377643 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:02.377260 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:43:02.979666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:02.979631 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85"] Oct 30 18:43:02.979939 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:02.979888 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" podUID="40dd342e-5dc7-4057-ac59-9ea1f8e244b7" containerName="tempo" containerID="cri-o://cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352" gracePeriod=30 Oct 30 18:43:03.983929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:03.983897 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp"] Oct 30 18:43:03.984314 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:03.984150 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" podUID="c00c4728-ce69-493a-859f-8aa4eb2161ec" containerName="tempo" containerID="cri-o://1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb" gracePeriod=30 Oct 30 18:43:04.027538 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.027508 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6"] Oct 30 18:43:04.027900 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.027736 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" podUID="f2129aca-0d65-412e-bc81-f90e40f10eb3" containerName="tempo" containerID="cri-o://47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411" gracePeriod=30 Oct 30 18:43:04.153795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.153742 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:43:04.211556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.211527 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-conf\") pod \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " Oct 30 18:43:04.211736 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.211589 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-kube-api-access-w899x\" (UniqueName: \"kubernetes.io/projected/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-kube-api-access-w899x\") pod \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " Oct 30 18:43:04.211736 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.211665 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-ca-bundle\") pod \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " Oct 30 18:43:04.211876 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.211735 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-tmp-storage\") pod \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " Oct 30 18:43:04.211876 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.211771 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-distributor-mtls\") pod \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\" (UID: \"40dd342e-5dc7-4057-ac59-9ea1f8e244b7\") " Oct 30 18:43:04.212395 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.212041 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-tmp-storage" pod "40dd342e-5dc7-4057-ac59-9ea1f8e244b7" (UID: "40dd342e-5dc7-4057-ac59-9ea1f8e244b7"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:04.212395 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.212083 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-conf" pod "40dd342e-5dc7-4057-ac59-9ea1f8e244b7" (UID: "40dd342e-5dc7-4057-ac59-9ea1f8e244b7"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:04.212395 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.212120 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-ca-bundle" pod "40dd342e-5dc7-4057-ac59-9ea1f8e244b7" (UID: "40dd342e-5dc7-4057-ac59-9ea1f8e244b7"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:04.213989 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.213966 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-distributor-mtls" pod "40dd342e-5dc7-4057-ac59-9ea1f8e244b7" (UID: "40dd342e-5dc7-4057-ac59-9ea1f8e244b7"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:04.214092 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.214078 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-kube-api-access-w899x" pod "40dd342e-5dc7-4057-ac59-9ea1f8e244b7" (UID: "40dd342e-5dc7-4057-ac59-9ea1f8e244b7"). InnerVolumeSpecName "kube-api-access-w899x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:04.313216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.313191 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:04.313216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.313216 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-tmp-storage\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:04.313349 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.313226 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:04.313349 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.313237 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:04.313349 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.313246 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w899x\" (UniqueName: \"kubernetes.io/projected/40dd342e-5dc7-4057-ac59-9ea1f8e244b7-kube-api-access-w899x\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:04.391309 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.391279 5014 generic.go:356] "Generic (PLEG): container finished" podID="40dd342e-5dc7-4057-ac59-9ea1f8e244b7" containerID="cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352" exitCode=0 Oct 30 18:43:04.391399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.391324 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" Oct 30 18:43:04.391399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.391331 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" event={"ID":"40dd342e-5dc7-4057-ac59-9ea1f8e244b7","Type":"ContainerDied","Data":"cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352"} Oct 30 18:43:04.391399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.391381 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85" event={"ID":"40dd342e-5dc7-4057-ac59-9ea1f8e244b7","Type":"ContainerDied","Data":"0b9f8c84c6f887dadbb8c097bf8aed0a66c09beada622bbf5763b95b9992ac56"} Oct 30 18:43:04.391505 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.391409 5014 scope.go:122] "RemoveContainer" containerID="cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352" Oct 30 18:43:04.400627 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.400586 5014 scope.go:122] "RemoveContainer" containerID="cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352" Oct 30 18:43:04.400885 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:43:04.400867 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352\": container with ID starting with cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352 not found: ID does not exist" containerID="cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352" Oct 30 18:43:04.400928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.400894 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352"} err="failed to get container status \"cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352\": rpc error: code = NotFound desc = could not find container \"cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352\": container with ID starting with cb703a3aa64828cea303f3b42a386b8343f34b1d64533fc60cc948bed446e352 not found: ID does not exist" Oct 30 18:43:04.441035 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.440984 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85"] Oct 30 18:43:04.445500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:04.445481 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-9644b84bf-mnq85"] Oct 30 18:43:05.141147 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:05.141112 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40dd342e-5dc7-4057-ac59-9ea1f8e244b7" path="/var/lib/kubelet/pods/40dd342e-5dc7-4057-ac59-9ea1f8e244b7/volumes" Oct 30 18:43:07.307053 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:07.307016 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="4be88b2b-0140-4e66-bfc6-2ab58e64e57d" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:12.276165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:12.276127 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" podUID="c00c4728-ce69-493a-859f-8aa4eb2161ec" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:12.276688 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:12.276497 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" podUID="27f44482-f8d8-4e8c-9dab-6fae485ca581" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:12.377336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:12.377310 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" podUID="f2129aca-0d65-412e-bc81-f90e40f10eb3" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:13.333482 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:43:13.333446 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:43:17.305409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:17.305372 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="4be88b2b-0140-4e66-bfc6-2ab58e64e57d" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:22.276270 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:22.276215 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" podUID="c00c4728-ce69-493a-859f-8aa4eb2161ec" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:22.276771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:22.276653 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" podUID="27f44482-f8d8-4e8c-9dab-6fae485ca581" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:22.477282 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:22.477251 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" podUID="f2129aca-0d65-412e-bc81-f90e40f10eb3" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:27.305600 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:27.305571 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:43:32.276258 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:32.276218 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" podUID="c00c4728-ce69-493a-859f-8aa4eb2161ec" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:32.276748 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:32.276319 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:43:32.276960 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:32.276935 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" podUID="27f44482-f8d8-4e8c-9dab-6fae485ca581" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:32.375177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:32.375150 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" podUID="f2129aca-0d65-412e-bc81-f90e40f10eb3" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:32.375344 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:32.375242 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:43:34.150214 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.150192 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:43:34.187803 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.187782 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:43:34.245690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.245628 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-storage-gcs-key\") pod \"c00c4728-ce69-493a-859f-8aa4eb2161ec\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " Oct 30 18:43:34.245690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.245666 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-conf\") pod \"c00c4728-ce69-493a-859f-8aa4eb2161ec\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " Oct 30 18:43:34.245909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.245699 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-querier-mtls\") pod \"c00c4728-ce69-493a-859f-8aa4eb2161ec\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " Oct 30 18:43:34.245909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.245804 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-ca-bundle\") pod \"c00c4728-ce69-493a-859f-8aa4eb2161ec\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " Oct 30 18:43:34.245909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.245903 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-tmp-storage\") pod \"c00c4728-ce69-493a-859f-8aa4eb2161ec\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " Oct 30 18:43:34.246077 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.245957 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00c4728-ce69-493a-859f-8aa4eb2161ec-kube-api-access-l77b4\" (UniqueName: \"kubernetes.io/projected/c00c4728-ce69-493a-859f-8aa4eb2161ec-kube-api-access-l77b4\") pod \"c00c4728-ce69-493a-859f-8aa4eb2161ec\" (UID: \"c00c4728-ce69-493a-859f-8aa4eb2161ec\") " Oct 30 18:43:34.246171 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.246148 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-conf" pod "c00c4728-ce69-493a-859f-8aa4eb2161ec" (UID: "c00c4728-ce69-493a-859f-8aa4eb2161ec"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:34.246244 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.246187 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-ca-bundle" pod "c00c4728-ce69-493a-859f-8aa4eb2161ec" (UID: "c00c4728-ce69-493a-859f-8aa4eb2161ec"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:34.246375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.246354 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.246431 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.246383 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.246828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.246806 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-tmp-storage" pod "c00c4728-ce69-493a-859f-8aa4eb2161ec" (UID: "c00c4728-ce69-493a-859f-8aa4eb2161ec"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:34.248382 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.248351 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-querier-mtls" pod "c00c4728-ce69-493a-859f-8aa4eb2161ec" (UID: "c00c4728-ce69-493a-859f-8aa4eb2161ec"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:34.248578 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.248555 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-storage-gcs-key" pod "c00c4728-ce69-493a-859f-8aa4eb2161ec" (UID: "c00c4728-ce69-493a-859f-8aa4eb2161ec"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:34.248842 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.248814 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00c4728-ce69-493a-859f-8aa4eb2161ec-kube-api-access-l77b4" pod "c00c4728-ce69-493a-859f-8aa4eb2161ec" (UID: "c00c4728-ce69-493a-859f-8aa4eb2161ec"). InnerVolumeSpecName "kube-api-access-l77b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:34.346646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.346620 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2129aca-0d65-412e-bc81-f90e40f10eb3-kube-api-access-tt85c\" (UniqueName: \"kubernetes.io/projected/f2129aca-0d65-412e-bc81-f90e40f10eb3-kube-api-access-tt85c\") pod \"f2129aca-0d65-412e-bc81-f90e40f10eb3\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " Oct 30 18:43:34.346771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.346687 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-storage-gcs-key\") pod \"f2129aca-0d65-412e-bc81-f90e40f10eb3\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " Oct 30 18:43:34.346771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.346758 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-query-frontend-mtls\") pod \"f2129aca-0d65-412e-bc81-f90e40f10eb3\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " Oct 30 18:43:34.346893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.346796 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-tmp-storage\") pod \"f2129aca-0d65-412e-bc81-f90e40f10eb3\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " Oct 30 18:43:34.346893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.346833 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-ca-bundle\") pod \"f2129aca-0d65-412e-bc81-f90e40f10eb3\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " Oct 30 18:43:34.347004 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.346890 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-conf\") pod \"f2129aca-0d65-412e-bc81-f90e40f10eb3\" (UID: \"f2129aca-0d65-412e-bc81-f90e40f10eb3\") " Oct 30 18:43:34.347142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.347126 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-tmp-storage\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.347208 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.347150 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l77b4\" (UniqueName: \"kubernetes.io/projected/c00c4728-ce69-493a-859f-8aa4eb2161ec-kube-api-access-l77b4\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.347208 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.347164 5014 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-storage-gcs-key\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.347208 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.347168 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-tmp-storage" pod "f2129aca-0d65-412e-bc81-f90e40f10eb3" (UID: "f2129aca-0d65-412e-bc81-f90e40f10eb3"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:34.347208 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.347179 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/c00c4728-ce69-493a-859f-8aa4eb2161ec-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.347345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.347226 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-ca-bundle" pod "f2129aca-0d65-412e-bc81-f90e40f10eb3" (UID: "f2129aca-0d65-412e-bc81-f90e40f10eb3"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:34.347345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.347279 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-conf" pod "f2129aca-0d65-412e-bc81-f90e40f10eb3" (UID: "f2129aca-0d65-412e-bc81-f90e40f10eb3"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:34.348993 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.348971 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-query-frontend-mtls" pod "f2129aca-0d65-412e-bc81-f90e40f10eb3" (UID: "f2129aca-0d65-412e-bc81-f90e40f10eb3"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:34.349249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.349224 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-storage-gcs-key" pod "f2129aca-0d65-412e-bc81-f90e40f10eb3" (UID: "f2129aca-0d65-412e-bc81-f90e40f10eb3"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:34.349317 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.349284 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2129aca-0d65-412e-bc81-f90e40f10eb3-kube-api-access-tt85c" pod "f2129aca-0d65-412e-bc81-f90e40f10eb3" (UID: "f2129aca-0d65-412e-bc81-f90e40f10eb3"). InnerVolumeSpecName "kube-api-access-tt85c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:34.447714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.447694 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.447714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.447714 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt85c\" (UniqueName: \"kubernetes.io/projected/f2129aca-0d65-412e-bc81-f90e40f10eb3-kube-api-access-tt85c\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.447836 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.447724 5014 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-storage-gcs-key\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.447836 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.447734 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.447836 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.447743 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-tmp-storage\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.447836 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.447754 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2129aca-0d65-412e-bc81-f90e40f10eb3-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.535844 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.535783 5014 generic.go:356] "Generic (PLEG): container finished" podID="f2129aca-0d65-412e-bc81-f90e40f10eb3" containerID="47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411" exitCode=137 Oct 30 18:43:34.535844 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.535827 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" Oct 30 18:43:34.535986 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.535870 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" event={"ID":"f2129aca-0d65-412e-bc81-f90e40f10eb3","Type":"ContainerDied","Data":"47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411"} Oct 30 18:43:34.535986 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.535953 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6" event={"ID":"f2129aca-0d65-412e-bc81-f90e40f10eb3","Type":"ContainerDied","Data":"e26b896fffb3e4fb5b188aa754d1ae628c068253e467e9a399ee6d4081eb73b5"} Oct 30 18:43:34.535986 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.535982 5014 scope.go:122] "RemoveContainer" containerID="47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411" Oct 30 18:43:34.537280 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.537258 5014 generic.go:356] "Generic (PLEG): container finished" podID="c00c4728-ce69-493a-859f-8aa4eb2161ec" containerID="1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb" exitCode=137 Oct 30 18:43:34.537375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.537329 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" Oct 30 18:43:34.537375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.537349 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" event={"ID":"c00c4728-ce69-493a-859f-8aa4eb2161ec","Type":"ContainerDied","Data":"1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb"} Oct 30 18:43:34.537469 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.537401 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp" event={"ID":"c00c4728-ce69-493a-859f-8aa4eb2161ec","Type":"ContainerDied","Data":"06fd4228f9c019d8b08debd58b85354736f7ed9b24ba0f184460a2944208b00f"} Oct 30 18:43:34.545723 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.545708 5014 scope.go:122] "RemoveContainer" containerID="47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411" Oct 30 18:43:34.545955 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:43:34.545940 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411\": container with ID starting with 47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411 not found: ID does not exist" containerID="47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411" Oct 30 18:43:34.546005 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.545963 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411"} err="failed to get container status \"47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411\": rpc error: code = NotFound desc = could not find container \"47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411\": container with ID starting with 47722b717f0dc4df6ec2c261b298edf149537fdcdf0f89dc7f92f85ecbd9d411 not found: ID does not exist" Oct 30 18:43:34.546005 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.545979 5014 scope.go:122] "RemoveContainer" containerID="1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb" Oct 30 18:43:34.553901 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.553883 5014 scope.go:122] "RemoveContainer" containerID="1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb" Oct 30 18:43:34.554145 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:43:34.554130 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb\": container with ID starting with 1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb not found: ID does not exist" containerID="1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb" Oct 30 18:43:34.554196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.554151 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb"} err="failed to get container status \"1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb\": rpc error: code = NotFound desc = could not find container \"1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb\": container with ID starting with 1963deb70ed7676b43e85431cd1e1cd5907afcd121a10f611dd8e5aef0062ffb not found: ID does not exist" Oct 30 18:43:34.586204 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.586177 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6"] Oct 30 18:43:34.589810 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.589791 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75944dc8fb-zvls6"] Oct 30 18:43:34.623007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.622986 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp"] Oct 30 18:43:34.626718 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:34.626699 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-687b9b5c79-87cvp"] Oct 30 18:43:35.141126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:35.141101 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00c4728-ce69-493a-859f-8aa4eb2161ec" path="/var/lib/kubelet/pods/c00c4728-ce69-493a-859f-8aa4eb2161ec/volumes" Oct 30 18:43:35.141494 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:35.141481 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2129aca-0d65-412e-bc81-f90e40f10eb3" path="/var/lib/kubelet/pods/f2129aca-0d65-412e-bc81-f90e40f10eb3/volumes" Oct 30 18:43:42.276714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:42.276678 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" podUID="27f44482-f8d8-4e8c-9dab-6fae485ca581" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:43.332743 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:43:43.332708 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:43:43.981683 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:43.981646 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5"] Oct 30 18:43:43.981962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:43.981913 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" podUID="27f44482-f8d8-4e8c-9dab-6fae485ca581" containerName="tempo" containerID="cri-o://ab293791839df0ded9ef871fc667ace32751077e3ce8b2936a77ef0440551c5a" gracePeriod=30 Oct 30 18:43:43.986179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:43.986135 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" podUID="27f44482-f8d8-4e8c-9dab-6fae485ca581" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:45.601954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.601926 5014 generic.go:356] "Generic (PLEG): container finished" podID="27f44482-f8d8-4e8c-9dab-6fae485ca581" containerID="ab293791839df0ded9ef871fc667ace32751077e3ce8b2936a77ef0440551c5a" exitCode=0 Oct 30 18:43:45.602239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.601995 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" event={"ID":"27f44482-f8d8-4e8c-9dab-6fae485ca581","Type":"ContainerDied","Data":"ab293791839df0ded9ef871fc667ace32751077e3ce8b2936a77ef0440551c5a"} Oct 30 18:43:45.739054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.739023 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:43:45.828949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.828906 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-tmp-storage\") pod \"27f44482-f8d8-4e8c-9dab-6fae485ca581\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " Oct 30 18:43:45.828949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.828956 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-ca-bundle\") pod \"27f44482-f8d8-4e8c-9dab-6fae485ca581\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " Oct 30 18:43:45.829163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.828996 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-compactor-mtls\") pod \"27f44482-f8d8-4e8c-9dab-6fae485ca581\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " Oct 30 18:43:45.829163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.829021 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f44482-f8d8-4e8c-9dab-6fae485ca581-kube-api-access-kphzr\" (UniqueName: \"kubernetes.io/projected/27f44482-f8d8-4e8c-9dab-6fae485ca581-kube-api-access-kphzr\") pod \"27f44482-f8d8-4e8c-9dab-6fae485ca581\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " Oct 30 18:43:45.829163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.829046 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-storage-gcs-key\") pod \"27f44482-f8d8-4e8c-9dab-6fae485ca581\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " Oct 30 18:43:45.829163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.829067 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-conf\") pod \"27f44482-f8d8-4e8c-9dab-6fae485ca581\" (UID: \"27f44482-f8d8-4e8c-9dab-6fae485ca581\") " Oct 30 18:43:45.829384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.829227 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-tmp-storage" pod "27f44482-f8d8-4e8c-9dab-6fae485ca581" (UID: "27f44482-f8d8-4e8c-9dab-6fae485ca581"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:45.829462 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.829436 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-ca-bundle" pod "27f44482-f8d8-4e8c-9dab-6fae485ca581" (UID: "27f44482-f8d8-4e8c-9dab-6fae485ca581"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:45.829536 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.829499 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-conf" pod "27f44482-f8d8-4e8c-9dab-6fae485ca581" (UID: "27f44482-f8d8-4e8c-9dab-6fae485ca581"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:45.831322 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.831297 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-compactor-mtls" pod "27f44482-f8d8-4e8c-9dab-6fae485ca581" (UID: "27f44482-f8d8-4e8c-9dab-6fae485ca581"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:45.831430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.831395 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-storage-gcs-key" pod "27f44482-f8d8-4e8c-9dab-6fae485ca581" (UID: "27f44482-f8d8-4e8c-9dab-6fae485ca581"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:45.831430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.831393 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f44482-f8d8-4e8c-9dab-6fae485ca581-kube-api-access-kphzr" pod "27f44482-f8d8-4e8c-9dab-6fae485ca581" (UID: "27f44482-f8d8-4e8c-9dab-6fae485ca581"). InnerVolumeSpecName "kube-api-access-kphzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:45.929618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.929568 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:45.929618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.929611 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kphzr\" (UniqueName: \"kubernetes.io/projected/27f44482-f8d8-4e8c-9dab-6fae485ca581-kube-api-access-kphzr\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:45.929776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.929627 5014 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/27f44482-f8d8-4e8c-9dab-6fae485ca581-storage-gcs-key\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:45.929776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.929637 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:45.929776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.929644 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-tmp-storage\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:45.929776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:45.929652 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27f44482-f8d8-4e8c-9dab-6fae485ca581-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:46.607557 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:46.607530 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" Oct 30 18:43:46.607557 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:46.607545 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5" event={"ID":"27f44482-f8d8-4e8c-9dab-6fae485ca581","Type":"ContainerDied","Data":"349e41aec40a06a500cfe6c193a31725ddac53c9dc48519b54f92601eca3baa7"} Oct 30 18:43:46.608056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:46.607622 5014 scope.go:122] "RemoveContainer" containerID="ab293791839df0ded9ef871fc667ace32751077e3ce8b2936a77ef0440551c5a" Oct 30 18:43:46.657330 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:46.657306 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5"] Oct 30 18:43:46.662332 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:46.662310 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-854c845df5-cmhx5"] Oct 30 18:43:47.141410 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:47.141377 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f44482-f8d8-4e8c-9dab-6fae485ca581" path="/var/lib/kubelet/pods/27f44482-f8d8-4e8c-9dab-6fae485ca581/volumes" Oct 30 18:43:49.295638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.295571 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:43:49.296259 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.295854 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="461bf434-3215-4c93-9095-adc018d78854" containerName="otc-container" containerID="cri-o://7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20" gracePeriod=30 Oct 30 18:43:49.552233 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.552213 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:49.623319 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.623284 5014 generic.go:356] "Generic (PLEG): container finished" podID="461bf434-3215-4c93-9095-adc018d78854" containerID="7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20" exitCode=0 Oct 30 18:43:49.623473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.623324 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:49.623473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.623358 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"461bf434-3215-4c93-9095-adc018d78854","Type":"ContainerDied","Data":"7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20"} Oct 30 18:43:49.623473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.623408 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"461bf434-3215-4c93-9095-adc018d78854","Type":"ContainerDied","Data":"a3be19b9634b9a97e9c187d511521c9d61fb64b633b054e03db25ad33a9f8bd8"} Oct 30 18:43:49.623473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.623430 5014 scope.go:122] "RemoveContainer" containerID="7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20" Oct 30 18:43:49.633402 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.633386 5014 scope.go:122] "RemoveContainer" containerID="7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20" Oct 30 18:43:49.633696 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:43:49.633678 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20\": container with ID starting with 7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20 not found: ID does not exist" containerID="7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20" Oct 30 18:43:49.633778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.633709 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20"} err="failed to get container status \"7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20\": rpc error: code = NotFound desc = could not find container \"7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20\": container with ID starting with 7a67a385d3a07fadf1add2450d26491a1b1cc1313fb57b8c339d223312caeb20 not found: ID does not exist" Oct 30 18:43:49.653066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.653047 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-monitoring-tls-certs\") pod \"461bf434-3215-4c93-9095-adc018d78854\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " Oct 30 18:43:49.653179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.653109 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/461bf434-3215-4c93-9095-adc018d78854-otc-internal\" (UniqueName: \"kubernetes.io/configmap/461bf434-3215-4c93-9095-adc018d78854-otc-internal\") pod \"461bf434-3215-4c93-9095-adc018d78854\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " Oct 30 18:43:49.653179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.653147 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461bf434-3215-4c93-9095-adc018d78854-kube-api-access-zndj5\" (UniqueName: \"kubernetes.io/projected/461bf434-3215-4c93-9095-adc018d78854-kube-api-access-zndj5\") pod \"461bf434-3215-4c93-9095-adc018d78854\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " Oct 30 18:43:49.653292 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.653176 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-otlp-tls-certs\") pod \"461bf434-3215-4c93-9095-adc018d78854\" (UID: \"461bf434-3215-4c93-9095-adc018d78854\") " Oct 30 18:43:49.653511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.653486 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461bf434-3215-4c93-9095-adc018d78854-otc-internal" pod "461bf434-3215-4c93-9095-adc018d78854" (UID: "461bf434-3215-4c93-9095-adc018d78854"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:49.655146 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.655121 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-monitoring-tls-certs" pod "461bf434-3215-4c93-9095-adc018d78854" (UID: "461bf434-3215-4c93-9095-adc018d78854"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:49.655283 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.655264 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461bf434-3215-4c93-9095-adc018d78854-kube-api-access-zndj5" pod "461bf434-3215-4c93-9095-adc018d78854" (UID: "461bf434-3215-4c93-9095-adc018d78854"). InnerVolumeSpecName "kube-api-access-zndj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:49.655377 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.655322 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-otlp-tls-certs" pod "461bf434-3215-4c93-9095-adc018d78854" (UID: "461bf434-3215-4c93-9095-adc018d78854"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:49.753893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.753871 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/461bf434-3215-4c93-9095-adc018d78854-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:49.753893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.753892 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zndj5\" (UniqueName: \"kubernetes.io/projected/461bf434-3215-4c93-9095-adc018d78854-kube-api-access-zndj5\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:49.754025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.753902 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:49.754025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.753911 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/461bf434-3215-4c93-9095-adc018d78854-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:43:49.967084 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.967051 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:43:49.978528 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:49.978506 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:43:50.006263 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006235 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:43:50.006620 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006574 5014 cpu_manager.go:414] "containerMap: removing container" podUID="461bf434-3215-4c93-9095-adc018d78854" containerName="otc-container" Oct 30 18:43:50.006620 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006609 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461bf434-3215-4c93-9095-adc018d78854" containerName="otc-container" Oct 30 18:43:50.006761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006634 5014 cpu_manager.go:414] "containerMap: removing container" podUID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" containerName="tempo-gateway" Oct 30 18:43:50.006761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006642 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" containerName="tempo-gateway" Oct 30 18:43:50.006761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006664 5014 cpu_manager.go:414] "containerMap: removing container" podUID="f2129aca-0d65-412e-bc81-f90e40f10eb3" containerName="tempo" Oct 30 18:43:50.006761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006672 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2129aca-0d65-412e-bc81-f90e40f10eb3" containerName="tempo" Oct 30 18:43:50.006761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006685 5014 cpu_manager.go:414] "containerMap: removing container" podUID="c00c4728-ce69-493a-859f-8aa4eb2161ec" containerName="tempo" Oct 30 18:43:50.006761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006693 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00c4728-ce69-493a-859f-8aa4eb2161ec" containerName="tempo" Oct 30 18:43:50.006761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006707 5014 cpu_manager.go:414] "containerMap: removing container" podUID="40dd342e-5dc7-4057-ac59-9ea1f8e244b7" containerName="tempo" Oct 30 18:43:50.006761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006715 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40dd342e-5dc7-4057-ac59-9ea1f8e244b7" containerName="tempo" Oct 30 18:43:50.006761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006735 5014 cpu_manager.go:414] "containerMap: removing container" podUID="27f44482-f8d8-4e8c-9dab-6fae485ca581" containerName="tempo" Oct 30 18:43:50.006761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006743 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f44482-f8d8-4e8c-9dab-6fae485ca581" containerName="tempo" Oct 30 18:43:50.006761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006760 5014 cpu_manager.go:414] "containerMap: removing container" podUID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" containerName="tempo-gateway-opa" Oct 30 18:43:50.007285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006767 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" containerName="tempo-gateway-opa" Oct 30 18:43:50.007285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006862 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="461bf434-3215-4c93-9095-adc018d78854" containerName="otc-container" Oct 30 18:43:50.007285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006877 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" containerName="tempo-gateway" Oct 30 18:43:50.007285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006889 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2129aca-0d65-412e-bc81-f90e40f10eb3" containerName="tempo" Oct 30 18:43:50.007285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006898 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00c4728-ce69-493a-859f-8aa4eb2161ec" containerName="tempo" Oct 30 18:43:50.007285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006906 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="40dd342e-5dc7-4057-ac59-9ea1f8e244b7" containerName="tempo" Oct 30 18:43:50.007285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006916 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f44482-f8d8-4e8c-9dab-6fae485ca581" containerName="tempo" Oct 30 18:43:50.007285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.006925 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f935b6e1-fc16-4cce-ad16-2cd7bc8cdb72" containerName="tempo-gateway-opa" Oct 30 18:43:50.011748 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.011730 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.026716 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.026694 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-qcbgc\"" Oct 30 18:43:50.026716 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.026706 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 18:43:50.026839 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.026708 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:43:50.026839 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.026763 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:43:50.031731 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.031635 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:43:50.056523 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.056498 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.056671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.056536 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.056671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.056636 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53350c40-cefe-46c2-9876-24a4ca540659-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.056812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.056691 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz2jz\" (UniqueName: \"kubernetes.io/projected/53350c40-cefe-46c2-9876-24a4ca540659-kube-api-access-tz2jz\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.157343 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.157320 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53350c40-cefe-46c2-9876-24a4ca540659-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.157450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.157378 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz2jz\" (UniqueName: \"kubernetes.io/projected/53350c40-cefe-46c2-9876-24a4ca540659-kube-api-access-tz2jz\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.157450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.157397 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.157632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.157576 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.158236 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.158176 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53350c40-cefe-46c2-9876-24a4ca540659-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.159835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.159819 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.159957 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.159941 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.180369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.180344 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz2jz\" (UniqueName: \"kubernetes.io/projected/53350c40-cefe-46c2-9876-24a4ca540659-kube-api-access-tz2jz\") pod \"data-science-collector-collector-0\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.323464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.323446 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:43:50.624675 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.624585 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:43:50.629723 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:43:50.629694 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53350c40_cefe_46c2_9876_24a4ca540659.slice/crio-b4f7675ed160475d7a79441f6da7951ae93674cc691e54926c5cb73c1f6d4257 WatchSource:0}: Error finding container b4f7675ed160475d7a79441f6da7951ae93674cc691e54926c5cb73c1f6d4257: Status 404 returned error can't find the container with id b4f7675ed160475d7a79441f6da7951ae93674cc691e54926c5cb73c1f6d4257 Oct 30 18:43:50.631772 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.631747 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:43:50.631847 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:50.631805 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:43:51.141545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:51.141510 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="461bf434-3215-4c93-9095-adc018d78854" path="/var/lib/kubelet/pods/461bf434-3215-4c93-9095-adc018d78854/volumes" Oct 30 18:43:51.635767 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:51.635732 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"53350c40-cefe-46c2-9876-24a4ca540659","Type":"ContainerStarted","Data":"6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8"} Oct 30 18:43:51.636275 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:51.635795 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"53350c40-cefe-46c2-9876-24a4ca540659","Type":"ContainerStarted","Data":"b4f7675ed160475d7a79441f6da7951ae93674cc691e54926c5cb73c1f6d4257"} Oct 30 18:43:51.680355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:43:51.680301 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.680289587 podStartE2EDuration="2.680289587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:51.678738486 +0000 UTC m=+2844.667078972" watchObservedRunningTime="2026-10-30 18:43:51.680289587 +0000 UTC m=+2844.668630074" Oct 30 18:44:13.332308 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:44:13.332270 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:44:20.480214 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.480179 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:20.480856 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.480474 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="53350c40-cefe-46c2-9876-24a4ca540659" containerName="otc-container" containerID="cri-o://6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8" gracePeriod=30 Oct 30 18:44:20.740355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.740275 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:20.767651 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.767626 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-monitoring-tls-certs\") pod \"53350c40-cefe-46c2-9876-24a4ca540659\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " Oct 30 18:44:20.767767 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.767662 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-otlp-tls-certs\") pod \"53350c40-cefe-46c2-9876-24a4ca540659\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " Oct 30 18:44:20.767767 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.767693 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53350c40-cefe-46c2-9876-24a4ca540659-kube-api-access-tz2jz\" (UniqueName: \"kubernetes.io/projected/53350c40-cefe-46c2-9876-24a4ca540659-kube-api-access-tz2jz\") pod \"53350c40-cefe-46c2-9876-24a4ca540659\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " Oct 30 18:44:20.767767 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.767747 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53350c40-cefe-46c2-9876-24a4ca540659-otc-internal\" (UniqueName: \"kubernetes.io/configmap/53350c40-cefe-46c2-9876-24a4ca540659-otc-internal\") pod \"53350c40-cefe-46c2-9876-24a4ca540659\" (UID: \"53350c40-cefe-46c2-9876-24a4ca540659\") " Oct 30 18:44:20.768166 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.768130 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53350c40-cefe-46c2-9876-24a4ca540659-otc-internal" pod "53350c40-cefe-46c2-9876-24a4ca540659" (UID: "53350c40-cefe-46c2-9876-24a4ca540659"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:20.769805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.769777 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-monitoring-tls-certs" pod "53350c40-cefe-46c2-9876-24a4ca540659" (UID: "53350c40-cefe-46c2-9876-24a4ca540659"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:20.769934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.769918 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53350c40-cefe-46c2-9876-24a4ca540659-kube-api-access-tz2jz" pod "53350c40-cefe-46c2-9876-24a4ca540659" (UID: "53350c40-cefe-46c2-9876-24a4ca540659"). InnerVolumeSpecName "kube-api-access-tz2jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:20.769984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.769926 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-otlp-tls-certs" pod "53350c40-cefe-46c2-9876-24a4ca540659" (UID: "53350c40-cefe-46c2-9876-24a4ca540659"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:20.779212 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.779185 5014 generic.go:356] "Generic (PLEG): container finished" podID="53350c40-cefe-46c2-9876-24a4ca540659" containerID="6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8" exitCode=0 Oct 30 18:44:20.779336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.779245 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:20.779336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.779276 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"53350c40-cefe-46c2-9876-24a4ca540659","Type":"ContainerDied","Data":"6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8"} Oct 30 18:44:20.779336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.779328 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"53350c40-cefe-46c2-9876-24a4ca540659","Type":"ContainerDied","Data":"b4f7675ed160475d7a79441f6da7951ae93674cc691e54926c5cb73c1f6d4257"} Oct 30 18:44:20.779483 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.779358 5014 scope.go:122] "RemoveContainer" containerID="6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8" Oct 30 18:44:20.796335 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.796296 5014 scope.go:122] "RemoveContainer" containerID="6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8" Oct 30 18:44:20.796675 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:44:20.796641 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8\": container with ID starting with 6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8 not found: ID does not exist" containerID="6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8" Oct 30 18:44:20.796762 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.796683 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8"} err="failed to get container status \"6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8\": rpc error: code = NotFound desc = could not find container \"6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8\": container with ID starting with 6630a4b04d7b499f2565e0074b13993b905fb835a07393e5c5d4cd41ba9555b8 not found: ID does not exist" Oct 30 18:44:20.831007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.830987 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:20.835853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.835832 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:20.869112 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.869087 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:20.869112 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.869109 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53350c40-cefe-46c2-9876-24a4ca540659-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:20.869241 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.869119 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz2jz\" (UniqueName: \"kubernetes.io/projected/53350c40-cefe-46c2-9876-24a4ca540659-kube-api-access-tz2jz\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:20.869241 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:20.869126 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53350c40-cefe-46c2-9876-24a4ca540659-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:21.024503 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:21.024413 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:44:21.024699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:21.024670 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="4be88b2b-0140-4e66-bfc6-2ab58e64e57d" containerName="tempo" containerID="cri-o://1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798" gracePeriod=30 Oct 30 18:44:21.141802 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:21.141773 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53350c40-cefe-46c2-9876-24a4ca540659" path="/var/lib/kubelet/pods/53350c40-cefe-46c2-9876-24a4ca540659/volumes" Oct 30 18:44:21.707661 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:21.707625 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"] Oct 30 18:44:21.712240 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:21.712210 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"] Oct 30 18:44:23.141331 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:23.141296 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd" path="/var/lib/kubelet/pods/caa96ec4-aabd-4ba6-8aa9-ace3b533e4dd/volumes" Oct 30 18:44:25.535082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.535054 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:25.535464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.535363 5014 cpu_manager.go:414] "containerMap: removing container" podUID="53350c40-cefe-46c2-9876-24a4ca540659" containerName="otc-container" Oct 30 18:44:25.535464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.535374 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53350c40-cefe-46c2-9876-24a4ca540659" containerName="otc-container" Oct 30 18:44:25.535626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.535470 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="53350c40-cefe-46c2-9876-24a4ca540659" containerName="otc-container" Oct 30 18:44:25.538762 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.538743 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.556177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.555996 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:44:25.558626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.558582 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:44:25.564559 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.564537 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7zmp2\"" Oct 30 18:44:25.567737 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.567714 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 18:44:25.569824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.569801 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:25.603511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.603477 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.603680 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.603631 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6h88\" (UniqueName: \"kubernetes.io/projected/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-kube-api-access-f6h88\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.603754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.603679 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.603754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.603726 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.704526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.704496 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.704684 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.704542 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6h88\" (UniqueName: \"kubernetes.io/projected/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-kube-api-access-f6h88\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.704684 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.704561 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.704684 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.704582 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.705161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.705141 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.707173 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.707151 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.707298 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.707194 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.728018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.727996 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6h88\" (UniqueName: \"kubernetes.io/projected/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-kube-api-access-f6h88\") pod \"data-science-collector-collector-0\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:25.851841 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:25.851819 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:26.053275 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.053252 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:44:26.108405 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.108346 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ingester-mtls\") pod \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " Oct 30 18:44:26.108405 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.108404 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-storage-gcs-key\") pod \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " Oct 30 18:44:26.108617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.108429 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ca-bundle\") pod \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " Oct 30 18:44:26.108617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.108532 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") pod \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " Oct 30 18:44:26.108617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.108571 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-kube-api-access-b4s8q\" (UniqueName: \"kubernetes.io/projected/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-kube-api-access-b4s8q\") pod \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " Oct 30 18:44:26.108818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.108633 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-conf\") pod \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\" (UID: \"4be88b2b-0140-4e66-bfc6-2ab58e64e57d\") " Oct 30 18:44:26.109304 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.109262 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-conf" pod "4be88b2b-0140-4e66-bfc6-2ab58e64e57d" (UID: "4be88b2b-0140-4e66-bfc6-2ab58e64e57d"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:26.109420 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.109309 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ca-bundle" pod "4be88b2b-0140-4e66-bfc6-2ab58e64e57d" (UID: "4be88b2b-0140-4e66-bfc6-2ab58e64e57d"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:26.110893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.110870 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ingester-mtls" pod "4be88b2b-0140-4e66-bfc6-2ab58e64e57d" (UID: "4be88b2b-0140-4e66-bfc6-2ab58e64e57d"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:26.111251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.111236 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-storage-gcs-key" pod "4be88b2b-0140-4e66-bfc6-2ab58e64e57d" (UID: "4be88b2b-0140-4e66-bfc6-2ab58e64e57d"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:26.111478 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.111450 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-kube-api-access-b4s8q" pod "4be88b2b-0140-4e66-bfc6-2ab58e64e57d" (UID: "4be88b2b-0140-4e66-bfc6-2ab58e64e57d"). InnerVolumeSpecName "kube-api-access-b4s8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:26.112257 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.112237 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1" pod "4be88b2b-0140-4e66-bfc6-2ab58e64e57d" (UID: "4be88b2b-0140-4e66-bfc6-2ab58e64e57d"). InnerVolumeSpecName "pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:44:26.152830 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.152807 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:26.157573 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:44:26.157551 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65029fe8_c0f3_4368_8dba_7d4a8fb5a364.slice/crio-6e1029512748b0d19083b08ba608686b705f607b9ac5e0be5c17f50d099d8289 WatchSource:0}: Error finding container 6e1029512748b0d19083b08ba608686b705f607b9ac5e0be5c17f50d099d8289: Status 404 returned error can't find the container with id 6e1029512748b0d19083b08ba608686b705f607b9ac5e0be5c17f50d099d8289 Oct 30 18:44:26.159439 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.159412 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:44:26.159522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.159473 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:44:26.209525 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.209507 5014 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-storage-gcs-key\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:26.209634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.209527 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:26.209634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.209555 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:44:26.209634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.209565 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4s8q\" (UniqueName: \"kubernetes.io/projected/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-kube-api-access-b4s8q\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:26.209634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.209574 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:26.209634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.209582 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/4be88b2b-0140-4e66-bfc6-2ab58e64e57d-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:26.223698 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.223683 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:44:26.310664 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.310630 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-bdbe3e53-a4b1-49c9-98bb-7aefc61dafe2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a419edf7aaeacd1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-2de5aa6ecd623838765e0f31befbb65b2679c3ca7e322e1ba1051ac031e970ac\"" Oct 30 18:44:26.395487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.395125 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:44:26.395712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.395647 5014 cpu_manager.go:414] "containerMap: removing container" podUID="4be88b2b-0140-4e66-bfc6-2ab58e64e57d" containerName="tempo" Oct 30 18:44:26.395712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.395666 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be88b2b-0140-4e66-bfc6-2ab58e64e57d" containerName="tempo" Oct 30 18:44:26.395898 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.395834 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be88b2b-0140-4e66-bfc6-2ab58e64e57d" containerName="tempo" Oct 30 18:44:26.401855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.401476 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.416046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.416010 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:44:26.416281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.416259 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:44:26.416452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.416316 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:44:26.416452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.416373 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:44:26.416452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.416375 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:44:26.416452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.416436 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-k4b7c\"" Oct 30 18:44:26.418575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.418555 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:44:26.418702 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.418684 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:44:26.429837 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.429804 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:44:26.613120 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.613093 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.613486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.613126 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.613486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.613216 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.613486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.613270 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.613486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.613329 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.613486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.613375 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.613486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.613446 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.613486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.613464 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5czh\" (UniqueName: \"kubernetes.io/projected/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-kube-api-access-d5czh\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.714480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.714457 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.714552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.714514 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.714552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.714543 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.714672 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.714585 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.714672 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.714647 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.714784 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.714672 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5czh\" (UniqueName: \"kubernetes.io/projected/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-kube-api-access-d5czh\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.714784 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.714711 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.714784 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.714742 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.715350 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.715292 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.715350 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.715292 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.716255 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.716054 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.716255 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.716171 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.718135 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.718113 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.718414 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.718391 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.718925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.718902 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.737962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.737941 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5czh\" (UniqueName: \"kubernetes.io/projected/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-kube-api-access-d5czh\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:26.811008 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.810982 5014 generic.go:356] "Generic (PLEG): container finished" podID="4be88b2b-0140-4e66-bfc6-2ab58e64e57d" containerID="1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798" exitCode=0 Oct 30 18:44:26.811108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.811027 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 18:44:26.811108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.811065 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"4be88b2b-0140-4e66-bfc6-2ab58e64e57d","Type":"ContainerDied","Data":"1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798"} Oct 30 18:44:26.811206 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.811122 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"4be88b2b-0140-4e66-bfc6-2ab58e64e57d","Type":"ContainerDied","Data":"a0f6e91df190332fbb2f1661167c7cb77243fbe6a2052858901b0a134558aee6"} Oct 30 18:44:26.811206 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.811144 5014 scope.go:122] "RemoveContainer" containerID="1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798" Oct 30 18:44:26.813208 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.813072 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"65029fe8-c0f3-4368-8dba-7d4a8fb5a364","Type":"ContainerStarted","Data":"b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234"} Oct 30 18:44:26.813208 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.813119 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"65029fe8-c0f3-4368-8dba-7d4a8fb5a364","Type":"ContainerStarted","Data":"6e1029512748b0d19083b08ba608686b705f607b9ac5e0be5c17f50d099d8289"} Oct 30 18:44:26.822095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.822081 5014 scope.go:122] "RemoveContainer" containerID="1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798" Oct 30 18:44:26.822332 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:44:26.822310 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798\": container with ID starting with 1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798 not found: ID does not exist" containerID="1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798" Oct 30 18:44:26.822414 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.822337 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798"} err="failed to get container status \"1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798\": rpc error: code = NotFound desc = could not find container \"1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798\": container with ID starting with 1028d7be892be46a866f84699cfb9938f58b08781b8575f5d911c6b04200a798 not found: ID does not exist" Oct 30 18:44:26.864274 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.864248 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:44:26.873071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.873041 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 18:44:26.912700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:26.912659 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.9126485579999999 podStartE2EDuration="1.912648558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:26.911794116 +0000 UTC m=+2879.900134615" watchObservedRunningTime="2026-10-30 18:44:26.912648558 +0000 UTC m=+2879.900989044" Oct 30 18:44:27.141354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:27.141326 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be88b2b-0140-4e66-bfc6-2ab58e64e57d" path="/var/lib/kubelet/pods/4be88b2b-0140-4e66-bfc6-2ab58e64e57d/volumes" Oct 30 18:44:28.729927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:28.729895 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:28.759991 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:28.759965 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:28.830544 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:28.830517 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:28.830692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:28.830644 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:28.844288 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:28.844258 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:28.945450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:28.945421 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:28.978685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:28.978662 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:29.132446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.132414 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-k4b7c\"" Oct 30 18:44:29.139364 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.139335 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:29.469143 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.469062 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:44:29.473035 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:44:29.473005 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f250ef9_2cac_4f58_9d62_1ea89cb27d92.slice/crio-9210851b9c914e7fea5bc1418082ce12d4d66d65220f385a6bfd825b8422b1e2 WatchSource:0}: Error finding container 9210851b9c914e7fea5bc1418082ce12d4d66d65220f385a6bfd825b8422b1e2: Status 404 returned error can't find the container with id 9210851b9c914e7fea5bc1418082ce12d4d66d65220f385a6bfd825b8422b1e2 Oct 30 18:44:29.827355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.827324 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f250ef9-2cac-4f58-9d62-1ea89cb27d92","Type":"ContainerStarted","Data":"dc4c3a7a8ee1be719363386a22bfde54a54094cef0e881caead4a6112f0ef634"} Oct 30 18:44:29.827795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.827373 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f250ef9-2cac-4f58-9d62-1ea89cb27d92","Type":"ContainerStarted","Data":"cb4616a94e32b8222d7f3c375ee60ac4d6377d0b29e64a2a386a7703ce2f078b"} Oct 30 18:44:29.827795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.827392 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f250ef9-2cac-4f58-9d62-1ea89cb27d92","Type":"ContainerStarted","Data":"03029218a53a6228775fb9bb73e89e54438d3303597f621bf78196d365b3f519"} Oct 30 18:44:29.827795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.827409 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f250ef9-2cac-4f58-9d62-1ea89cb27d92","Type":"ContainerStarted","Data":"9210851b9c914e7fea5bc1418082ce12d4d66d65220f385a6bfd825b8422b1e2"} Oct 30 18:44:29.827795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.827433 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:29.827795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.827450 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:29.830125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.830104 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:29.835931 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.835915 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:29.874148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:29.874105 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=3.874093206 podStartE2EDuration="3.874093206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:29.872636824 +0000 UTC m=+2882.860977312" watchObservedRunningTime="2026-10-30 18:44:29.874093206 +0000 UTC m=+2882.862433693" Oct 30 18:44:30.833336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:30.833302 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:43.229805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.229768 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:43.231348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.230006 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="65029fe8-c0f3-4368-8dba-7d4a8fb5a364" containerName="otc-container" containerID="cri-o://b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234" gracePeriod=30 Oct 30 18:44:43.332954 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:44:43.332911 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:44:43.490569 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.490512 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:43.544686 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.544658 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-monitoring-tls-certs\") pod \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " Oct 30 18:44:43.544845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.544695 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-kube-api-access-f6h88\" (UniqueName: \"kubernetes.io/projected/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-kube-api-access-f6h88\") pod \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " Oct 30 18:44:43.544845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.544727 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otc-internal\" (UniqueName: \"kubernetes.io/configmap/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otc-internal\") pod \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " Oct 30 18:44:43.544845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.544776 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otlp-tls-certs\") pod \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\" (UID: \"65029fe8-c0f3-4368-8dba-7d4a8fb5a364\") " Oct 30 18:44:43.545170 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.545136 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otc-internal" pod "65029fe8-c0f3-4368-8dba-7d4a8fb5a364" (UID: "65029fe8-c0f3-4368-8dba-7d4a8fb5a364"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.547079 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.547053 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-monitoring-tls-certs" pod "65029fe8-c0f3-4368-8dba-7d4a8fb5a364" (UID: "65029fe8-c0f3-4368-8dba-7d4a8fb5a364"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:43.548142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.547891 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otlp-tls-certs" pod "65029fe8-c0f3-4368-8dba-7d4a8fb5a364" (UID: "65029fe8-c0f3-4368-8dba-7d4a8fb5a364"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:43.548381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.548346 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-kube-api-access-f6h88" pod "65029fe8-c0f3-4368-8dba-7d4a8fb5a364" (UID: "65029fe8-c0f3-4368-8dba-7d4a8fb5a364"). InnerVolumeSpecName "kube-api-access-f6h88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.645817 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.645789 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.645817 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.645814 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.645981 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.645824 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6h88\" (UniqueName: \"kubernetes.io/projected/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-kube-api-access-f6h88\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.645981 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.645833 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/65029fe8-c0f3-4368-8dba-7d4a8fb5a364-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.674495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.674469 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:44:43.675169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.674741 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo" containerID="cri-o://03029218a53a6228775fb9bb73e89e54438d3303597f621bf78196d365b3f519" gracePeriod=30 Oct 30 18:44:43.675395 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.674831 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo-gateway-opa" containerID="cri-o://dc4c3a7a8ee1be719363386a22bfde54a54094cef0e881caead4a6112f0ef634" gracePeriod=30 Oct 30 18:44:43.675495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.674833 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo-gateway" containerID="cri-o://cb4616a94e32b8222d7f3c375ee60ac4d6377d0b29e64a2a386a7703ce2f078b" gracePeriod=30 Oct 30 18:44:43.892176 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.892151 5014 generic.go:356] "Generic (PLEG): container finished" podID="65029fe8-c0f3-4368-8dba-7d4a8fb5a364" containerID="b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234" exitCode=0 Oct 30 18:44:43.892289 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.892196 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:43.892289 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.892240 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"65029fe8-c0f3-4368-8dba-7d4a8fb5a364","Type":"ContainerDied","Data":"b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234"} Oct 30 18:44:43.892396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.892302 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"65029fe8-c0f3-4368-8dba-7d4a8fb5a364","Type":"ContainerDied","Data":"6e1029512748b0d19083b08ba608686b705f607b9ac5e0be5c17f50d099d8289"} Oct 30 18:44:43.892396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.892346 5014 scope.go:122] "RemoveContainer" containerID="b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234" Oct 30 18:44:43.894681 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.894662 5014 generic.go:356] "Generic (PLEG): container finished" podID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerID="dc4c3a7a8ee1be719363386a22bfde54a54094cef0e881caead4a6112f0ef634" exitCode=0 Oct 30 18:44:43.894681 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.894679 5014 generic.go:356] "Generic (PLEG): container finished" podID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerID="03029218a53a6228775fb9bb73e89e54438d3303597f621bf78196d365b3f519" exitCode=0 Oct 30 18:44:43.894805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.894732 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f250ef9-2cac-4f58-9d62-1ea89cb27d92","Type":"ContainerDied","Data":"dc4c3a7a8ee1be719363386a22bfde54a54094cef0e881caead4a6112f0ef634"} Oct 30 18:44:43.894805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.894786 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f250ef9-2cac-4f58-9d62-1ea89cb27d92","Type":"ContainerDied","Data":"03029218a53a6228775fb9bb73e89e54438d3303597f621bf78196d365b3f519"} Oct 30 18:44:43.900862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.900845 5014 scope.go:122] "RemoveContainer" containerID="b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234" Oct 30 18:44:43.901124 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:44:43.901106 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234\": container with ID starting with b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234 not found: ID does not exist" containerID="b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234" Oct 30 18:44:43.901165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.901133 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234"} err="failed to get container status \"b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234\": rpc error: code = NotFound desc = could not find container \"b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234\": container with ID starting with b2b1ab32bcf88fffdf3babda21c7fb6e8fc27d571dc92dfdd1ec47c2018df234 not found: ID does not exist" Oct 30 18:44:43.942265 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.942242 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:43.946144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:43.946125 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:44.828027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:44.827983 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.106:8083/ready\": dial tcp 10.132.0.106:8083: connect: connection refused" Oct 30 18:44:44.828417 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:44.827981 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.132.0.106:8081/ready\": dial tcp 10.132.0.106:8081: connect: connection refused" Oct 30 18:44:44.902655 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:44.902628 5014 generic.go:356] "Generic (PLEG): container finished" podID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerID="cb4616a94e32b8222d7f3c375ee60ac4d6377d0b29e64a2a386a7703ce2f078b" exitCode=0 Oct 30 18:44:44.902791 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:44.902704 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f250ef9-2cac-4f58-9d62-1ea89cb27d92","Type":"ContainerDied","Data":"cb4616a94e32b8222d7f3c375ee60ac4d6377d0b29e64a2a386a7703ce2f078b"} Oct 30 18:44:44.940997 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:44.940975 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:45.055241 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055217 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-ca-bundle\") pod \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " Oct 30 18:44:45.055347 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055324 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " Oct 30 18:44:45.055392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055357 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-tenants\") pod \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " Oct 30 18:44:45.055424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055390 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-rbac\") pod \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " Oct 30 18:44:45.055424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055418 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-kube-api-access-d5czh\" (UniqueName: \"kubernetes.io/projected/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-kube-api-access-d5czh\") pod \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " Oct 30 18:44:45.055658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055491 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-mtls\") pod \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " Oct 30 18:44:45.055658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055528 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-ca-bundle" pod "1f250ef9-2cac-4f58-9d62-1ea89cb27d92" (UID: "1f250ef9-2cac-4f58-9d62-1ea89cb27d92"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:45.055658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055551 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-conf\") pod \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " Oct 30 18:44:45.055658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055615 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " Oct 30 18:44:45.055658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055648 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\" (UID: \"1f250ef9-2cac-4f58-9d62-1ea89cb27d92\") " Oct 30 18:44:45.056108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055868 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:45.056108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055869 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-rbac" pod "1f250ef9-2cac-4f58-9d62-1ea89cb27d92" (UID: "1f250ef9-2cac-4f58-9d62-1ea89cb27d92"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:45.056108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.055997 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-conf" pod "1f250ef9-2cac-4f58-9d62-1ea89cb27d92" (UID: "1f250ef9-2cac-4f58-9d62-1ea89cb27d92"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:45.056108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.056008 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-serving-cabundle" pod "1f250ef9-2cac-4f58-9d62-1ea89cb27d92" (UID: "1f250ef9-2cac-4f58-9d62-1ea89cb27d92"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:45.057903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.057879 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-mtls" pod "1f250ef9-2cac-4f58-9d62-1ea89cb27d92" (UID: "1f250ef9-2cac-4f58-9d62-1ea89cb27d92"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:45.058018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.058000 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "1f250ef9-2cac-4f58-9d62-1ea89cb27d92" (UID: "1f250ef9-2cac-4f58-9d62-1ea89cb27d92"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:45.058097 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.058029 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-tenants" pod "1f250ef9-2cac-4f58-9d62-1ea89cb27d92" (UID: "1f250ef9-2cac-4f58-9d62-1ea89cb27d92"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:45.058454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.058436 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-kube-api-access-d5czh" pod "1f250ef9-2cac-4f58-9d62-1ea89cb27d92" (UID: "1f250ef9-2cac-4f58-9d62-1ea89cb27d92"). InnerVolumeSpecName "kube-api-access-d5czh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:45.058945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.058928 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "1f250ef9-2cac-4f58-9d62-1ea89cb27d92" (UID: "1f250ef9-2cac-4f58-9d62-1ea89cb27d92"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:44:45.141309 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.141284 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65029fe8-c0f3-4368-8dba-7d4a8fb5a364" path="/var/lib/kubelet/pods/65029fe8-c0f3-4368-8dba-7d4a8fb5a364/volumes" Oct 30 18:44:45.156672 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.156653 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:45.156776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.156675 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:45.156776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.156691 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:45.156776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.156705 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:45.156776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.156740 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:44:45.156776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.156755 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:45.156776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.156768 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-gateway-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:45.157088 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.156781 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5czh\" (UniqueName: \"kubernetes.io/projected/1f250ef9-2cac-4f58-9d62-1ea89cb27d92-kube-api-access-d5czh\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:44:45.165256 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.165242 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:44:45.257658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.257637 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" Oct 30 18:44:45.909617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.909581 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:44:45.909994 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.909622 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f250ef9-2cac-4f58-9d62-1ea89cb27d92","Type":"ContainerDied","Data":"9210851b9c914e7fea5bc1418082ce12d4d66d65220f385a6bfd825b8422b1e2"} Oct 30 18:44:45.909994 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.909685 5014 scope.go:122] "RemoveContainer" containerID="dc4c3a7a8ee1be719363386a22bfde54a54094cef0e881caead4a6112f0ef634" Oct 30 18:44:45.918478 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.918462 5014 scope.go:122] "RemoveContainer" containerID="cb4616a94e32b8222d7f3c375ee60ac4d6377d0b29e64a2a386a7703ce2f078b" Oct 30 18:44:45.926513 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.926490 5014 scope.go:122] "RemoveContainer" containerID="03029218a53a6228775fb9bb73e89e54438d3303597f621bf78196d365b3f519" Oct 30 18:44:45.971869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.971848 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:44:45.983944 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:45.983923 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:44:47.142700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:47.142658 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" path="/var/lib/kubelet/pods/1f250ef9-2cac-4f58-9d62-1ea89cb27d92/volumes" Oct 30 18:44:52.462475 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.462446 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2"] Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.462836 5014 cpu_manager.go:414] "containerMap: removing container" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo" Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.462852 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo" Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.462864 5014 cpu_manager.go:414] "containerMap: removing container" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo-gateway-opa" Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.462871 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo-gateway-opa" Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.462890 5014 cpu_manager.go:414] "containerMap: removing container" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo-gateway" Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.462899 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo-gateway" Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.462922 5014 cpu_manager.go:414] "containerMap: removing container" podUID="65029fe8-c0f3-4368-8dba-7d4a8fb5a364" containerName="otc-container" Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.462927 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65029fe8-c0f3-4368-8dba-7d4a8fb5a364" containerName="otc-container" Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.462994 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="65029fe8-c0f3-4368-8dba-7d4a8fb5a364" containerName="otc-container" Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.463005 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo" Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.463010 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo-gateway-opa" Oct 30 18:44:52.463050 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.463016 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f250ef9-2cac-4f58-9d62-1ea89cb27d92" containerName="tempo-gateway" Oct 30 18:44:52.467787 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.467767 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.484771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.484753 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:44:52.487894 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.487863 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:44:52.496770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.496748 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:44:52.499700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.499683 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 18:44:52.499796 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.499720 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:44:52.499844 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.499832 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-nqfzw\"" Oct 30 18:44:52.499899 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.499885 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 18:44:52.501415 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.501392 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2"] Oct 30 18:44:52.624040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.624015 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.624161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.624063 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.624161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.624127 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g757p\" (UniqueName: \"kubernetes.io/projected/e7797026-78dd-49a6-823d-e904cba88ab1-kube-api-access-g757p\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.624233 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.624178 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.624233 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.624206 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.725517 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.725452 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.725517 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.725486 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g757p\" (UniqueName: \"kubernetes.io/projected/e7797026-78dd-49a6-823d-e904cba88ab1-kube-api-access-g757p\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.725517 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.725510 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.725743 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.725571 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.725743 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.725650 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.725826 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:44:52.725772 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 18:44:52.725870 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:44:52.725844 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-client-cert podName:e7797026-78dd-49a6-823d-e904cba88ab1 nodeName:}" failed. No retries permitted until 2026-10-30 18:44:53.225826399 +0000 UTC m=+2906.214166885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" (UID: "e7797026-78dd-49a6-823d-e904cba88ab1") : secret "prometheus-operated-tls" not found Oct 30 18:44:52.726320 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.726297 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.728219 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.728194 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.728361 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.728255 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:52.749054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:52.749028 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g757p\" (UniqueName: \"kubernetes.io/projected/e7797026-78dd-49a6-823d-e904cba88ab1-kube-api-access-g757p\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:53.229764 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.229732 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:53.232274 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.232253 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hm7r2\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:53.380308 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.380284 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:53.676020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.675990 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2"] Oct 30 18:44:53.681627 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:44:53.681576 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7797026_78dd_49a6_823d_e904cba88ab1.slice/crio-cff53a8ea3aa183dab0e698aef43f95b0c7ff5853c8748cfa84666a29c11bc61 WatchSource:0}: Error finding container cff53a8ea3aa183dab0e698aef43f95b0c7ff5853c8748cfa84666a29c11bc61: Status 404 returned error can't find the container with id cff53a8ea3aa183dab0e698aef43f95b0c7ff5853c8748cfa84666a29c11bc61 Oct 30 18:44:53.897721 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.897668 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:53.902713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.902691 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:53.917540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.917514 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:44:53.929146 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.929069 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:44:53.929146 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.929102 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsglb\"" Oct 30 18:44:53.929346 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.929180 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:44:53.929346 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.929207 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:44:53.929346 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.929259 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:44:53.930923 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.930900 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:53.953078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:53.953041 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" event={"ID":"e7797026-78dd-49a6-823d-e904cba88ab1","Type":"ContainerStarted","Data":"cff53a8ea3aa183dab0e698aef43f95b0c7ff5853c8748cfa84666a29c11bc61"} Oct 30 18:44:54.036556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.036520 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.036736 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.036568 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.036736 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.036723 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.036840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.036792 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.036840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.036819 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.036840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.036837 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gzmw\" (UniqueName: \"kubernetes.io/projected/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-kube-api-access-4gzmw\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.138279 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.138247 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.138279 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.138280 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.138513 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.138309 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.138513 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.138341 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.138513 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.138364 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.138513 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.138382 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gzmw\" (UniqueName: \"kubernetes.io/projected/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-kube-api-access-4gzmw\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.138986 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.138963 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.140895 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.140869 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.140980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.140894 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.141326 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.141309 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.141405 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.141390 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.160799 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.160780 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gzmw\" (UniqueName: \"kubernetes.io/projected/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-kube-api-access-4gzmw\") pod \"data-science-collector-collector-0\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.218436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.218413 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:44:54.515168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.515098 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:44:54.520146 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:44:54.520110 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e0e559_ea2e_42d8_b13e_9b27b75dfecf.slice/crio-bd4eb6ae8ab54ec802c8a102afe4e1bd14b6c72ffd421fdf2dcd1d6b626fb875 WatchSource:0}: Error finding container bd4eb6ae8ab54ec802c8a102afe4e1bd14b6c72ffd421fdf2dcd1d6b626fb875: Status 404 returned error can't find the container with id bd4eb6ae8ab54ec802c8a102afe4e1bd14b6c72ffd421fdf2dcd1d6b626fb875 Oct 30 18:44:54.522428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.522396 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:44:54.522519 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.522460 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:44:54.959074 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.959036 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" event={"ID":"e7797026-78dd-49a6-823d-e904cba88ab1","Type":"ContainerStarted","Data":"90d634b1fb404380fe36a4f0be4123b49c8cfa458514434a3d98b469363cfd04"} Oct 30 18:44:54.959496 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.959108 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:44:54.960809 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.960787 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf","Type":"ContainerStarted","Data":"5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c"} Oct 30 18:44:54.960916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:54.960824 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf","Type":"ContainerStarted","Data":"bd4eb6ae8ab54ec802c8a102afe4e1bd14b6c72ffd421fdf2dcd1d6b626fb875"} Oct 30 18:44:55.003974 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:55.003932 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" podStartSLOduration=3.003918513 podStartE2EDuration="3.003918513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:55.002977132 +0000 UTC m=+2907.991317619" watchObservedRunningTime="2026-10-30 18:44:55.003918513 +0000 UTC m=+2907.992259000" Oct 30 18:44:55.048473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:44:55.048429 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.048415822 podStartE2EDuration="2.048415822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:55.046802328 +0000 UTC m=+2908.035142816" watchObservedRunningTime="2026-10-30 18:44:55.048415822 +0000 UTC m=+2908.036756309" Oct 30 18:45:00.711973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:00.711943 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6"] Oct 30 18:45:00.715584 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:00.715562 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" Oct 30 18:45:00.729840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:00.729820 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-5mtd6\"" Oct 30 18:45:00.742692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:00.742673 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6"] Oct 30 18:45:00.888824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:00.888798 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzq7p\" (UniqueName: \"kubernetes.io/projected/6aadb74c-066c-483b-9cea-f9114a5a596d-kube-api-access-nzq7p\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6\" (UID: \"6aadb74c-066c-483b-9cea-f9114a5a596d\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" Oct 30 18:45:00.968216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:00.968164 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:45:00.989380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:00.989354 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzq7p\" (UniqueName: \"kubernetes.io/projected/6aadb74c-066c-483b-9cea-f9114a5a596d-kube-api-access-nzq7p\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6\" (UID: \"6aadb74c-066c-483b-9cea-f9114a5a596d\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" Oct 30 18:45:01.026095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.026072 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzq7p\" (UniqueName: \"kubernetes.io/projected/6aadb74c-066c-483b-9cea-f9114a5a596d-kube-api-access-nzq7p\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6\" (UID: \"6aadb74c-066c-483b-9cea-f9114a5a596d\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" Oct 30 18:45:01.035734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.035714 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" Oct 30 18:45:01.339307 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.339282 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6"] Oct 30 18:45:01.344105 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:45:01.344076 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6aadb74c_066c_483b_9cea_f9114a5a596d.slice/crio-1636a8e3dc7eacb25e699e6592ca7690fb870ddf0ff66f0c7dbf07b651a2fbfc WatchSource:0}: Error finding container 1636a8e3dc7eacb25e699e6592ca7690fb870ddf0ff66f0c7dbf07b651a2fbfc: Status 404 returned error can't find the container with id 1636a8e3dc7eacb25e699e6592ca7690fb870ddf0ff66f0c7dbf07b651a2fbfc Oct 30 18:45:01.566282 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.566249 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:45:01.571719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.571695 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:45:01.571865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.571811 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.575848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.575827 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.586903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.586883 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:45:01.586903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.586896 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:45:01.587085 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.586906 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:45:01.587085 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.586895 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:45:01.587085 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.586964 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-q4sgp\"" Oct 30 18:45:01.591352 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.591310 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:45:01.594674 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.594658 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:45:01.594905 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.594888 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:45:01.594971 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.594922 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:45:01.599643 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.599622 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:45:01.608465 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.608442 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:45:01.686573 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.686546 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:45:01.686719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.686579 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:45:01.686806 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.686792 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:45:01.686872 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.686837 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:45:01.686918 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.686902 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-gjhkk\"" Oct 30 18:45:01.695404 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695374 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.695515 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695496 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.695629 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695537 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.695629 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695566 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.695740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695675 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.695740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695711 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.695833 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695774 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.695833 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695817 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.695948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695849 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.695948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695929 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.696039 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695961 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.696039 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.695991 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.696118 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.696052 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwzj9\" (UniqueName: \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-kube-api-access-dwzj9\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.696118 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.696079 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.696118 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.696096 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.696118 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.696115 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.696293 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.696136 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.696293 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.696162 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.696293 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.696196 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6nr\" (UniqueName: \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-kube-api-access-8b6nr\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.797492 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797461 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.797492 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797496 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797519 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797552 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwzj9\" (UniqueName: \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-kube-api-access-dwzj9\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797583 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797644 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797808 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797848 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797867 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797883 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6nr\" (UniqueName: \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-kube-api-access-8b6nr\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797918 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.797968 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.798004 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.798029 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.798028 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.798866 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.798066 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.798866 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.798093 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.798866 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.798145 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.798866 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.798172 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.798866 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.798203 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.799200 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.799123 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.800312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.799443 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.800312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.799960 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.800312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.800261 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.801115 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.801089 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.801303 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.801277 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.801902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.801842 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.801902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.801856 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.802369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.802350 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.803107 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.803018 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.803294 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.803272 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.803395 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.803308 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.803482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.803460 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.803482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.803475 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.803745 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.803725 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.804145 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.804121 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.804645 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.804623 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.820551 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.820525 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwzj9\" (UniqueName: \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-kube-api-access-dwzj9\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.821341 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.821319 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6nr\" (UniqueName: \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-kube-api-access-8b6nr\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:01.892506 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.892452 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:01.994289 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.994261 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" event={"ID":"6aadb74c-066c-483b-9cea-f9114a5a596d","Type":"ContainerStarted","Data":"83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050"} Oct 30 18:45:01.994427 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:01.994306 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" event={"ID":"6aadb74c-066c-483b-9cea-f9114a5a596d","Type":"ContainerStarted","Data":"1636a8e3dc7eacb25e699e6592ca7690fb870ddf0ff66f0c7dbf07b651a2fbfc"} Oct 30 18:45:02.032426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:02.032377 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" podStartSLOduration=2.032364331 podStartE2EDuration="2.032364331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:02.031417822 +0000 UTC m=+2915.019758308" watchObservedRunningTime="2026-10-30 18:45:02.032364331 +0000 UTC m=+2915.020704817" Oct 30 18:45:02.208962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:02.208804 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:45:02.212888 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:45:02.212857 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39107235_3fb6_49af_b295_9be59c67c288.slice/crio-105e9772e4840a15775360df7378b3bd98b2bf8073c92a4781758870d5a0194c WatchSource:0}: Error finding container 105e9772e4840a15775360df7378b3bd98b2bf8073c92a4781758870d5a0194c: Status 404 returned error can't find the container with id 105e9772e4840a15775360df7378b3bd98b2bf8073c92a4781758870d5a0194c Oct 30 18:45:03.001814 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:03.001778 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39107235-3fb6-49af-b295-9be59c67c288","Type":"ContainerStarted","Data":"a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb"} Oct 30 18:45:03.002355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:03.001844 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39107235-3fb6-49af-b295-9be59c67c288","Type":"ContainerStarted","Data":"105e9772e4840a15775360df7378b3bd98b2bf8073c92a4781758870d5a0194c"} Oct 30 18:45:06.017238 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:06.017160 5014 generic.go:356] "Generic (PLEG): container finished" podID="39107235-3fb6-49af-b295-9be59c67c288" containerID="a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb" exitCode=0 Oct 30 18:45:06.017574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:06.017235 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39107235-3fb6-49af-b295-9be59c67c288","Type":"ContainerDied","Data":"a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb"} Oct 30 18:45:07.024648 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:07.024570 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39107235-3fb6-49af-b295-9be59c67c288","Type":"ContainerStarted","Data":"df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3"} Oct 30 18:45:07.025072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:07.024702 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39107235-3fb6-49af-b295-9be59c67c288","Type":"ContainerStarted","Data":"fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19"} Oct 30 18:45:07.025072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:07.024922 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:07.095159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:07.095119 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.009499681 podStartE2EDuration="6.095108356s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.085608675s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:07.094194721 +0000 UTC m=+2920.082535208" watchObservedRunningTime="2026-10-30 18:45:07.095108356 +0000 UTC m=+2920.083448843" Oct 30 18:45:08.960194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:08.960158 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:08.988541 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:08.988512 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:09.061311 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:09.061272 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:09.061534 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:09.061444 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:09.075637 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:09.075584 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") DevicePath \"csi-3ece98a829a66f7cef6e6b4403002493a48d56f442e1673d82fff8fda237a009\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:09.178732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:09.178703 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3d0cf4484736f6d40d16a5331fa2b97b9730899f1120a88b3f09811f40fc91e0/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:09.205010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:09.204985 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:09.401964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:09.401934 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:09.748575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:09.748506 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:45:09.753788 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:45:09.753760 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1fa9ce_20a0_4578_9b1f_92c8f8ecaf7c.slice/crio-3d526c86d661743e31f4a61add01a6acb9478490c834aa8abecb3a5fab8df03d WatchSource:0}: Error finding container 3d526c86d661743e31f4a61add01a6acb9478490c834aa8abecb3a5fab8df03d: Status 404 returned error can't find the container with id 3d526c86d661743e31f4a61add01a6acb9478490c834aa8abecb3a5fab8df03d Oct 30 18:45:10.039959 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:10.039904 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c","Type":"ContainerStarted","Data":"3d526c86d661743e31f4a61add01a6acb9478490c834aa8abecb3a5fab8df03d"} Oct 30 18:45:11.053236 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:11.053198 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c","Type":"ContainerStarted","Data":"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9"} Oct 30 18:45:13.030692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:13.030655 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:45:13.331910 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:45:13.331872 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:45:14.069844 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:14.069813 5014 generic.go:356] "Generic (PLEG): container finished" podID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerID="03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9" exitCode=0 Oct 30 18:45:14.070195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:14.069865 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c","Type":"ContainerDied","Data":"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9"} Oct 30 18:45:15.076996 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:15.076965 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c","Type":"ContainerStarted","Data":"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884"} Oct 30 18:45:15.077381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:15.077059 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c","Type":"ContainerStarted","Data":"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432"} Oct 30 18:45:15.077381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:15.077083 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c","Type":"ContainerStarted","Data":"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc"} Oct 30 18:45:15.151426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:15.151384 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=10.66652755 podStartE2EDuration="14.151372616s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484845066s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:15.139558838 +0000 UTC m=+2928.127899336" watchObservedRunningTime="2026-10-30 18:45:15.151372616 +0000 UTC m=+2928.139713134" Oct 30 18:45:19.402405 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:19.402376 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:24.402628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:24.402581 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:24.407738 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:24.407717 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:25.130160 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:25.130131 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:45:42.723741 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:42.723703 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:45:42.724254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:42.723976 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" containerName="otc-container" containerID="cri-o://5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c" gracePeriod=30 Oct 30 18:45:43.333062 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:45:43.333030 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:45:45.079318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.079293 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.218028 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.217958 5014 generic.go:356] "Generic (PLEG): container finished" podID="c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" containerID="5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c" exitCode=0 Oct 30 18:45:45.218028 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.218002 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.218219 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.218057 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf","Type":"ContainerDied","Data":"5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c"} Oct 30 18:45:45.218219 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.218109 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf","Type":"ContainerDied","Data":"bd4eb6ae8ab54ec802c8a102afe4e1bd14b6c72ffd421fdf2dcd1d6b626fb875"} Oct 30 18:45:45.218219 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.218138 5014 scope.go:122] "RemoveContainer" containerID="5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c" Oct 30 18:45:45.227215 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.227200 5014 scope.go:122] "RemoveContainer" containerID="5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c" Oct 30 18:45:45.227477 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:45:45.227460 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c\": container with ID starting with 5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c not found: ID does not exist" containerID="5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c" Oct 30 18:45:45.227528 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.227487 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c"} err="failed to get container status \"5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c\": rpc error: code = NotFound desc = could not find container \"5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c\": container with ID starting with 5f4277220a07cc2d05bed2c41677bc499dc5fa42797b8c14b7910f20b7f1664c not found: ID does not exist" Oct 30 18:45:45.250432 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.250411 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otlp-tls-certs\") pod \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " Oct 30 18:45:45.250545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.250441 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otc-internal\") pod \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " Oct 30 18:45:45.250545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.250465 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-monitoring-tls-certs\") pod \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " Oct 30 18:45:45.250545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.250504 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-kube-api-access-4gzmw\" (UniqueName: \"kubernetes.io/projected/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-kube-api-access-4gzmw\") pod \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " Oct 30 18:45:45.250724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.250549 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-data-science-collector-ta-client-cert\") pod \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " Oct 30 18:45:45.250724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.250621 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-tls-certs\") pod \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\" (UID: \"c5e0e559-ea2e-42d8-b13e-9b27b75dfecf\") " Oct 30 18:45:45.251168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.250873 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otc-internal" pod "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" (UID: "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:45.252974 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.252943 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-tls-certs" pod "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" (UID: "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:45.253077 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.252982 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-kube-api-access-4gzmw" pod "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" (UID: "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf"). InnerVolumeSpecName "kube-api-access-4gzmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:45.253244 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.253218 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otlp-tls-certs" pod "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" (UID: "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:45.253510 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.253487 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-data-science-collector-ta-client-cert" pod "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" (UID: "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:45.253630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.253513 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-monitoring-tls-certs" pod "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" (UID: "c5e0e559-ea2e-42d8-b13e-9b27b75dfecf"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:45.351916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.351883 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:45:45.351916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.351913 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:45:45.351916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.351923 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:45:45.352080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.351932 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gzmw\" (UniqueName: \"kubernetes.io/projected/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-kube-api-access-4gzmw\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:45:45.352080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.351941 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:45:45.352080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.351951 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:45:45.569587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.569557 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:45:45.581247 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.581224 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:45:45.610504 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.610479 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:45:45.610859 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.610847 5014 cpu_manager.go:414] "containerMap: removing container" podUID="c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" containerName="otc-container" Oct 30 18:45:45.610941 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.610860 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" containerName="otc-container" Oct 30 18:45:45.610941 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.610940 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" containerName="otc-container" Oct 30 18:45:45.615423 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.615406 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.629782 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.629759 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:45:45.629877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.629776 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsglb\"" Oct 30 18:45:45.629877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.629865 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:45:45.629967 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.629866 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:45:45.629967 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.629896 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:45:45.629967 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.629943 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:45:45.635403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.635380 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:45:45.755100 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.755079 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.755221 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.755109 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.755221 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.755133 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.755221 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.755148 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.755351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.755278 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9gfv\" (UniqueName: \"kubernetes.io/projected/ab7d2910-7b19-43a6-b29f-e9ba8e517939-kube-api-access-z9gfv\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.755351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.755305 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.856507 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.856402 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.856507 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.856431 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.856507 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.856452 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.856507 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.856471 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.856841 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.856654 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9gfv\" (UniqueName: \"kubernetes.io/projected/ab7d2910-7b19-43a6-b29f-e9ba8e517939-kube-api-access-z9gfv\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.856841 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.856712 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.857249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.857205 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.859145 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.859124 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.859228 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.859147 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.859586 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.859565 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.859687 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.859655 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.879528 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.879505 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9gfv\" (UniqueName: \"kubernetes.io/projected/ab7d2910-7b19-43a6-b29f-e9ba8e517939-kube-api-access-z9gfv\") pod \"data-science-collector-collector-0\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:45.927446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:45.927424 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:45:46.225245 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:46.225177 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:45:46.229376 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:45:46.229349 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7d2910_7b19_43a6_b29f_e9ba8e517939.slice/crio-8b9df82b6ed9f6361fde1b5e4f5b33cab13250879822c120693e47a1b79bb25e WatchSource:0}: Error finding container 8b9df82b6ed9f6361fde1b5e4f5b33cab13250879822c120693e47a1b79bb25e: Status 404 returned error can't find the container with id 8b9df82b6ed9f6361fde1b5e4f5b33cab13250879822c120693e47a1b79bb25e Oct 30 18:45:46.231457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:46.231429 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:45:46.231532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:46.231490 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:45:47.142169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:47.142135 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e0e559-ea2e-42d8-b13e-9b27b75dfecf" path="/var/lib/kubelet/pods/c5e0e559-ea2e-42d8-b13e-9b27b75dfecf/volumes" Oct 30 18:45:47.230937 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:47.230902 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ab7d2910-7b19-43a6-b29f-e9ba8e517939","Type":"ContainerStarted","Data":"67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340"} Oct 30 18:45:47.231287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:47.230960 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ab7d2910-7b19-43a6-b29f-e9ba8e517939","Type":"ContainerStarted","Data":"8b9df82b6ed9f6361fde1b5e4f5b33cab13250879822c120693e47a1b79bb25e"} Oct 30 18:45:47.274040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:45:47.273993 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.273977378 podStartE2EDuration="2.273977378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:47.272189667 +0000 UTC m=+2960.260530155" watchObservedRunningTime="2026-10-30 18:45:47.273977378 +0000 UTC m=+2960.262317866" Oct 30 18:46:13.335860 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:46:13.335814 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:46:25.432911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.432874 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:46:25.438385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.438361 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.458460 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.458435 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:46:25.458690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.458671 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:46:25.467493 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.467462 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:46:25.467949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.467794 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:46:25.467949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.467810 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:46:25.467949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.467846 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:46:25.467949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.467914 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:46:25.467949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.467924 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-rjvl8\"" Oct 30 18:46:25.471026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.471004 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:46:25.565780 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.565744 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9gt\" (UniqueName: \"kubernetes.io/projected/d89d4050-7b48-41a6-adc0-6114c0793384-kube-api-access-dm9gt\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.565991 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.565787 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.565991 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.565820 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.565991 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.565970 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.566141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.566040 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.566141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.566079 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.566141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.566127 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.566257 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.566145 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.667177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.667142 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.667369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.667200 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.667369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.667227 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.667369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.667256 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.667369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.667278 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.667369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.667321 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9gt\" (UniqueName: \"kubernetes.io/projected/d89d4050-7b48-41a6-adc0-6114c0793384-kube-api-access-dm9gt\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.667369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.667358 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.667715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.667404 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.668381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.668352 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.668381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.668370 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.668588 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.668357 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.668839 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.668793 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.670128 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.670095 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.670284 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.670100 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.670284 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.670187 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:25.691744 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:25.691674 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9gt\" (UniqueName: \"kubernetes.io/projected/d89d4050-7b48-41a6-adc0-6114c0793384-kube-api-access-dm9gt\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:27.418134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:27.418028 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:46:27.426396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:27.426370 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:46:27.948835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:27.948804 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:27.949246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:27.949215 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="ab7d2910-7b19-43a6-b29f-e9ba8e517939" containerName="otc-container" containerID="cri-o://67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340" gracePeriod=30 Oct 30 18:46:30.300065 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.300040 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.407439 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.407412 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-tls-certs\") pod \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " Oct 30 18:46:30.407623 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.407446 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otc-internal\") pod \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " Oct 30 18:46:30.407623 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.407503 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otlp-tls-certs\") pod \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " Oct 30 18:46:30.407623 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.407549 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7d2910-7b19-43a6-b29f-e9ba8e517939-kube-api-access-z9gfv\" (UniqueName: \"kubernetes.io/projected/ab7d2910-7b19-43a6-b29f-e9ba8e517939-kube-api-access-z9gfv\") pod \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " Oct 30 18:46:30.407623 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.407589 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-monitoring-tls-certs\") pod \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " Oct 30 18:46:30.407910 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.407639 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-data-science-collector-ta-client-cert\") pod \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\" (UID: \"ab7d2910-7b19-43a6-b29f-e9ba8e517939\") " Oct 30 18:46:30.407998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.407972 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otc-internal" pod "ab7d2910-7b19-43a6-b29f-e9ba8e517939" (UID: "ab7d2910-7b19-43a6-b29f-e9ba8e517939"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:30.409812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.409781 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-tls-certs" pod "ab7d2910-7b19-43a6-b29f-e9ba8e517939" (UID: "ab7d2910-7b19-43a6-b29f-e9ba8e517939"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:30.409916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.409878 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7d2910-7b19-43a6-b29f-e9ba8e517939-kube-api-access-z9gfv" pod "ab7d2910-7b19-43a6-b29f-e9ba8e517939" (UID: "ab7d2910-7b19-43a6-b29f-e9ba8e517939"). InnerVolumeSpecName "kube-api-access-z9gfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:30.409980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.409965 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-data-science-collector-ta-client-cert" pod "ab7d2910-7b19-43a6-b29f-e9ba8e517939" (UID: "ab7d2910-7b19-43a6-b29f-e9ba8e517939"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:30.410336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.410309 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otlp-tls-certs" pod "ab7d2910-7b19-43a6-b29f-e9ba8e517939" (UID: "ab7d2910-7b19-43a6-b29f-e9ba8e517939"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:30.410336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.410324 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-monitoring-tls-certs" pod "ab7d2910-7b19-43a6-b29f-e9ba8e517939" (UID: "ab7d2910-7b19-43a6-b29f-e9ba8e517939"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:30.429569 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.429544 5014 generic.go:356] "Generic (PLEG): container finished" podID="ab7d2910-7b19-43a6-b29f-e9ba8e517939" containerID="67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340" exitCode=0 Oct 30 18:46:30.429671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.429589 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.429671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.429639 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ab7d2910-7b19-43a6-b29f-e9ba8e517939","Type":"ContainerDied","Data":"67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340"} Oct 30 18:46:30.429744 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.429692 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ab7d2910-7b19-43a6-b29f-e9ba8e517939","Type":"ContainerDied","Data":"8b9df82b6ed9f6361fde1b5e4f5b33cab13250879822c120693e47a1b79bb25e"} Oct 30 18:46:30.429744 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.429715 5014 scope.go:122] "RemoveContainer" containerID="67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340" Oct 30 18:46:30.438718 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.438661 5014 scope.go:122] "RemoveContainer" containerID="67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340" Oct 30 18:46:30.438971 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:46:30.438954 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340\": container with ID starting with 67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340 not found: ID does not exist" containerID="67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340" Oct 30 18:46:30.439019 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.438979 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340"} err="failed to get container status \"67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340\": rpc error: code = NotFound desc = could not find container \"67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340\": container with ID starting with 67fce509498df18e3012d3b4762677b367228467a4a947aff16f0997cced1340 not found: ID does not exist" Oct 30 18:46:30.480108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.480088 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:30.485807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.485780 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:30.508965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.508945 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:30.508965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.508965 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9gfv\" (UniqueName: \"kubernetes.io/projected/ab7d2910-7b19-43a6-b29f-e9ba8e517939-kube-api-access-z9gfv\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:30.509086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.508975 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:30.509086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.508985 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:30.509086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.508993 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d2910-7b19-43a6-b29f-e9ba8e517939-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:30.509086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.509002 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab7d2910-7b19-43a6-b29f-e9ba8e517939-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:30.912900 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.912872 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:30.941020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:30.940993 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:31.013802 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:31.013775 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:31.013922 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:31.013864 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:31.027873 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:31.027852 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:31.127548 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:31.127516 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:31.141886 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:31.141862 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7d2910-7b19-43a6-b29f-e9ba8e517939" path="/var/lib/kubelet/pods/ab7d2910-7b19-43a6-b29f-e9ba8e517939/volumes" Oct 30 18:46:31.154428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:31.154408 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:31.471779 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:31.471755 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-rjvl8\"" Oct 30 18:46:31.478849 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:31.478827 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:31.806797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:31.806765 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:46:31.810619 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:46:31.810578 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89d4050_7b48_41a6_adc0_6114c0793384.slice/crio-943b1d62711ff6a5816e48a17bb9fcc847ddf504a32e5821154a048b94842f21 WatchSource:0}: Error finding container 943b1d62711ff6a5816e48a17bb9fcc847ddf504a32e5821154a048b94842f21: Status 404 returned error can't find the container with id 943b1d62711ff6a5816e48a17bb9fcc847ddf504a32e5821154a048b94842f21 Oct 30 18:46:32.443873 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:32.443839 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d89d4050-7b48-41a6-adc0-6114c0793384","Type":"ContainerStarted","Data":"1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f"} Oct 30 18:46:32.444045 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:32.443899 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d89d4050-7b48-41a6-adc0-6114c0793384","Type":"ContainerStarted","Data":"f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6"} Oct 30 18:46:32.444091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:32.444033 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:32.444124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:32.444091 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:32.444162 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:32.444114 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d89d4050-7b48-41a6-adc0-6114c0793384","Type":"ContainerStarted","Data":"76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3"} Oct 30 18:46:32.444162 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:32.444150 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:32.444228 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:32.444166 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d89d4050-7b48-41a6-adc0-6114c0793384","Type":"ContainerStarted","Data":"943b1d62711ff6a5816e48a17bb9fcc847ddf504a32e5821154a048b94842f21"} Oct 30 18:46:32.446773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:32.446752 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:32.453487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:32.453465 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:32.500344 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:32.500297 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=7.500285612 podStartE2EDuration="7.500285612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:32.489919848 +0000 UTC m=+3005.478260335" watchObservedRunningTime="2026-10-30 18:46:32.500285612 +0000 UTC m=+3005.488626099" Oct 30 18:46:42.084403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:42.084370 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:46:42.084874 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:42.084640 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo" containerID="cri-o://76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3" gracePeriod=30 Oct 30 18:46:42.084874 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:42.084703 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo-gateway" containerID="cri-o://f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6" gracePeriod=30 Oct 30 18:46:42.084874 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:42.084701 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo-gateway-opa" containerID="cri-o://1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f" gracePeriod=30 Oct 30 18:46:42.445083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:42.445002 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.113:8083/ready\": dial tcp 10.132.0.113:8083: connect: connection refused" Oct 30 18:46:42.448845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:42.448816 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:46:42.491376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:42.491352 5014 generic.go:356] "Generic (PLEG): container finished" podID="d89d4050-7b48-41a6-adc0-6114c0793384" containerID="1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f" exitCode=0 Oct 30 18:46:42.491376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:42.491373 5014 generic.go:356] "Generic (PLEG): container finished" podID="d89d4050-7b48-41a6-adc0-6114c0793384" containerID="76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3" exitCode=0 Oct 30 18:46:42.491518 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:42.491429 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d89d4050-7b48-41a6-adc0-6114c0793384","Type":"ContainerDied","Data":"1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f"} Oct 30 18:46:42.491518 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:42.491482 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d89d4050-7b48-41a6-adc0-6114c0793384","Type":"ContainerDied","Data":"76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3"} Oct 30 18:46:43.335653 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:46:43.335615 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:46:43.355987 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.355966 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:43.498436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.498357 5014 generic.go:356] "Generic (PLEG): container finished" podID="d89d4050-7b48-41a6-adc0-6114c0793384" containerID="f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6" exitCode=0 Oct 30 18:46:43.498436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.498416 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:43.498436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.498430 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d89d4050-7b48-41a6-adc0-6114c0793384","Type":"ContainerDied","Data":"f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6"} Oct 30 18:46:43.498749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.498475 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d89d4050-7b48-41a6-adc0-6114c0793384","Type":"ContainerDied","Data":"943b1d62711ff6a5816e48a17bb9fcc847ddf504a32e5821154a048b94842f21"} Oct 30 18:46:43.498749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.498498 5014 scope.go:122] "RemoveContainer" containerID="1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f" Oct 30 18:46:43.507531 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.507510 5014 scope.go:122] "RemoveContainer" containerID="f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6" Oct 30 18:46:43.515401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.515380 5014 scope.go:122] "RemoveContainer" containerID="76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3" Oct 30 18:46:43.520662 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.520639 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-tenants\") pod \"d89d4050-7b48-41a6-adc0-6114c0793384\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " Oct 30 18:46:43.520756 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.520682 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89d4050-7b48-41a6-adc0-6114c0793384-kube-api-access-dm9gt\" (UniqueName: \"kubernetes.io/projected/d89d4050-7b48-41a6-adc0-6114c0793384-kube-api-access-dm9gt\") pod \"d89d4050-7b48-41a6-adc0-6114c0793384\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " Oct 30 18:46:43.520756 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.520702 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"d89d4050-7b48-41a6-adc0-6114c0793384\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " Oct 30 18:46:43.520881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.520764 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-ca-bundle\") pod \"d89d4050-7b48-41a6-adc0-6114c0793384\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " Oct 30 18:46:43.520881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.520804 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-rbac\") pod \"d89d4050-7b48-41a6-adc0-6114c0793384\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " Oct 30 18:46:43.520881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.520833 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-conf\") pod \"d89d4050-7b48-41a6-adc0-6114c0793384\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " Oct 30 18:46:43.521096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.520880 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"d89d4050-7b48-41a6-adc0-6114c0793384\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " Oct 30 18:46:43.521096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.520995 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"d89d4050-7b48-41a6-adc0-6114c0793384\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " Oct 30 18:46:43.521096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.521034 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-mtls\") pod \"d89d4050-7b48-41a6-adc0-6114c0793384\" (UID: \"d89d4050-7b48-41a6-adc0-6114c0793384\") " Oct 30 18:46:43.521251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.521159 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-serving-cabundle" pod "d89d4050-7b48-41a6-adc0-6114c0793384" (UID: "d89d4050-7b48-41a6-adc0-6114c0793384"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:43.521251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.521213 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-ca-bundle" pod "d89d4050-7b48-41a6-adc0-6114c0793384" (UID: "d89d4050-7b48-41a6-adc0-6114c0793384"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:43.521251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.521222 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-rbac" pod "d89d4050-7b48-41a6-adc0-6114c0793384" (UID: "d89d4050-7b48-41a6-adc0-6114c0793384"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:43.522072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.521620 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-conf" pod "d89d4050-7b48-41a6-adc0-6114c0793384" (UID: "d89d4050-7b48-41a6-adc0-6114c0793384"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:43.522072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.521637 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:43.522072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.521664 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:43.522072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.521682 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:43.523325 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.523301 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89d4050-7b48-41a6-adc0-6114c0793384-kube-api-access-dm9gt" pod "d89d4050-7b48-41a6-adc0-6114c0793384" (UID: "d89d4050-7b48-41a6-adc0-6114c0793384"). InnerVolumeSpecName "kube-api-access-dm9gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:43.524626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.524451 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-tenants" pod "d89d4050-7b48-41a6-adc0-6114c0793384" (UID: "d89d4050-7b48-41a6-adc0-6114c0793384"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:43.524626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.524506 5014 scope.go:122] "RemoveContainer" containerID="1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f" Oct 30 18:46:43.524626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.524575 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-mtls" pod "d89d4050-7b48-41a6-adc0-6114c0793384" (UID: "d89d4050-7b48-41a6-adc0-6114c0793384"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:43.524836 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.524628 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "d89d4050-7b48-41a6-adc0-6114c0793384" (UID: "d89d4050-7b48-41a6-adc0-6114c0793384"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:43.524894 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.524866 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "d89d4050-7b48-41a6-adc0-6114c0793384" (UID: "d89d4050-7b48-41a6-adc0-6114c0793384"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:46:43.525024 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:46:43.525001 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f\": container with ID starting with 1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f not found: ID does not exist" containerID="1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f" Oct 30 18:46:43.525141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.525031 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f"} err="failed to get container status \"1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f\": rpc error: code = NotFound desc = could not find container \"1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f\": container with ID starting with 1bf55c7666f86cea068a36abf5c0f22a4128b6ff1d5d29388c855aa40b73e81f not found: ID does not exist" Oct 30 18:46:43.525141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.525056 5014 scope.go:122] "RemoveContainer" containerID="f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6" Oct 30 18:46:43.525373 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:46:43.525331 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6\": container with ID starting with f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6 not found: ID does not exist" containerID="f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6" Oct 30 18:46:43.525373 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.525354 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6"} err="failed to get container status \"f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6\": rpc error: code = NotFound desc = could not find container \"f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6\": container with ID starting with f6a838c12ef4dcaf67b4e1e7703d8e2dad4c5887d5a34e62b8090f45afdbe6d6 not found: ID does not exist" Oct 30 18:46:43.525373 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.525370 5014 scope.go:122] "RemoveContainer" containerID="76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3" Oct 30 18:46:43.525759 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:46:43.525736 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3\": container with ID starting with 76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3 not found: ID does not exist" containerID="76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3" Oct 30 18:46:43.525840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.525767 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3"} err="failed to get container status \"76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3\": rpc error: code = NotFound desc = could not find container \"76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3\": container with ID starting with 76b8f456db6c79a99c22f53569287be6e6e0016a019bba53d3086052b754c9a3 not found: ID does not exist" Oct 30 18:46:43.622728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.622701 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:43.622850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.622745 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:46:43.622850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.622762 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:43.622850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.622771 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/d89d4050-7b48-41a6-adc0-6114c0793384-gateway-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:43.622850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.622780 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm9gt\" (UniqueName: \"kubernetes.io/projected/d89d4050-7b48-41a6-adc0-6114c0793384-kube-api-access-dm9gt\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:43.622850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.622789 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d89d4050-7b48-41a6-adc0-6114c0793384-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:46:43.631671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.631656 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:46:43.723252 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.723227 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" Oct 30 18:46:43.843287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.843263 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:46:43.855348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:43.855324 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:46:45.141457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.141422 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" path="/var/lib/kubelet/pods/d89d4050-7b48-41a6-adc0-6114c0793384/volumes" Oct 30 18:46:45.357964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.357934 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:45.358330 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358304 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo-gateway-opa" Oct 30 18:46:45.358330 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358324 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo-gateway-opa" Oct 30 18:46:45.358498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358337 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo" Oct 30 18:46:45.358498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358342 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo" Oct 30 18:46:45.358498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358373 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo-gateway" Oct 30 18:46:45.358498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358380 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo-gateway" Oct 30 18:46:45.358498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358391 5014 cpu_manager.go:414] "containerMap: removing container" podUID="ab7d2910-7b19-43a6-b29f-e9ba8e517939" containerName="otc-container" Oct 30 18:46:45.358498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358396 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7d2910-7b19-43a6-b29f-e9ba8e517939" containerName="otc-container" Oct 30 18:46:45.358498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358470 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7d2910-7b19-43a6-b29f-e9ba8e517939" containerName="otc-container" Oct 30 18:46:45.358498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358483 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo-gateway-opa" Oct 30 18:46:45.358498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358489 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo" Oct 30 18:46:45.358498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.358497 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89d4050-7b48-41a6-adc0-6114c0793384" containerName="tempo-gateway" Oct 30 18:46:45.363065 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.363047 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.389385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.389350 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsglb\"" Oct 30 18:46:45.389489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.389399 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:46:45.389489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.389399 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:46:45.389489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.389456 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:46:45.389642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.389514 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:46:45.400777 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.400720 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:46:45.402733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.402679 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:45.536489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.536465 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.536631 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.536506 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.536631 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.536525 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwfnx\" (UniqueName: \"kubernetes.io/projected/1fd4d728-3c76-4cf8-a700-33bba4fa9893-kube-api-access-rwfnx\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.536745 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.536624 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.536745 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.536669 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.536833 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.536782 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.637929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.637902 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.638040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.637959 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.638040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.637996 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.638163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.638050 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.638163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.638075 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwfnx\" (UniqueName: \"kubernetes.io/projected/1fd4d728-3c76-4cf8-a700-33bba4fa9893-kube-api-access-rwfnx\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.638163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.638115 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.638796 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.638772 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.640660 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.640636 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.640765 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.640674 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.641045 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.641026 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.641107 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.641042 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.660887 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.660830 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwfnx\" (UniqueName: \"kubernetes.io/projected/1fd4d728-3c76-4cf8-a700-33bba4fa9893-kube-api-access-rwfnx\") pod \"data-science-collector-collector-0\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.675642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.675621 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.973055 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.972991 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:45.977762 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:46:45.977735 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd4d728_3c76_4cf8_a700_33bba4fa9893.slice/crio-c02e6b1c9850ffb6b52208331ea44bb081700b1986ef9ed745cc78fb47c9a26f WatchSource:0}: Error finding container c02e6b1c9850ffb6b52208331ea44bb081700b1986ef9ed745cc78fb47c9a26f: Status 404 returned error can't find the container with id c02e6b1c9850ffb6b52208331ea44bb081700b1986ef9ed745cc78fb47c9a26f Oct 30 18:46:45.979955 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.979922 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:46:45.980063 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:45.979991 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:46:46.515779 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:46.515745 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1fd4d728-3c76-4cf8-a700-33bba4fa9893","Type":"ContainerStarted","Data":"0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da"} Oct 30 18:46:46.516183 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:46.515802 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1fd4d728-3c76-4cf8-a700-33bba4fa9893","Type":"ContainerStarted","Data":"c02e6b1c9850ffb6b52208331ea44bb081700b1986ef9ed745cc78fb47c9a26f"} Oct 30 18:46:46.558786 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:46.558746 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.558734577 podStartE2EDuration="1.558734577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:46.557263395 +0000 UTC m=+3019.545603927" watchObservedRunningTime="2026-10-30 18:46:46.558734577 +0000 UTC m=+3019.547075063" Oct 30 18:46:59.549794 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.549757 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:46:59.554408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.554384 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.569485 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.569463 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:46:59.569624 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.569463 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:46:59.569686 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.569619 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:46:59.581401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.581379 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-q57zs\"" Oct 30 18:46:59.581401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.581390 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:46:59.581401 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.581394 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:46:59.581608 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.581425 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:46:59.581608 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.581465 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:46:59.583450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.583422 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:46:59.644842 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.644819 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.644979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.644862 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.644979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.644896 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.644979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.644951 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.645116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.645013 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84glx\" (UniqueName: \"kubernetes.io/projected/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-kube-api-access-84glx\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.645116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.645047 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.645116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.645085 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.645240 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.645133 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.745716 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.745688 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.745835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.745731 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.745835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.745769 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.745835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.745796 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.745835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.745824 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84glx\" (UniqueName: \"kubernetes.io/projected/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-kube-api-access-84glx\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.746068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.745860 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.746068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.745922 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.746068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.745950 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.746657 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.746630 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.746807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.746690 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.746934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.746911 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.747055 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.747033 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.748498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.748476 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.748578 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.748513 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.748640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.748586 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:46:59.769178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:46:59.769155 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84glx\" (UniqueName: \"kubernetes.io/projected/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-kube-api-access-84glx\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:00.097641 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:00.097587 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:00.097895 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:00.097863 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="1fd4d728-3c76-4cf8-a700-33bba4fa9893" containerName="otc-container" containerID="cri-o://0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da" gracePeriod=30 Oct 30 18:47:01.355734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.355707 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.460203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.460175 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-data-science-collector-ta-client-cert\") pod \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " Oct 30 18:47:01.460381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.460218 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-monitoring-tls-certs\") pod \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " Oct 30 18:47:01.460381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.460248 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otc-internal\") pod \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " Oct 30 18:47:01.460381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.460316 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-tls-certs\") pod \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " Oct 30 18:47:01.460558 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.460395 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd4d728-3c76-4cf8-a700-33bba4fa9893-kube-api-access-rwfnx\" (UniqueName: \"kubernetes.io/projected/1fd4d728-3c76-4cf8-a700-33bba4fa9893-kube-api-access-rwfnx\") pod \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " Oct 30 18:47:01.460558 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.460439 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otlp-tls-certs\") pod \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\" (UID: \"1fd4d728-3c76-4cf8-a700-33bba4fa9893\") " Oct 30 18:47:01.460767 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.460731 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otc-internal" pod "1fd4d728-3c76-4cf8-a700-33bba4fa9893" (UID: "1fd4d728-3c76-4cf8-a700-33bba4fa9893"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:01.462575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.462542 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-monitoring-tls-certs" pod "1fd4d728-3c76-4cf8-a700-33bba4fa9893" (UID: "1fd4d728-3c76-4cf8-a700-33bba4fa9893"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:01.462713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.462609 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-data-science-collector-ta-client-cert" pod "1fd4d728-3c76-4cf8-a700-33bba4fa9893" (UID: "1fd4d728-3c76-4cf8-a700-33bba4fa9893"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:01.462713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.462644 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otlp-tls-certs" pod "1fd4d728-3c76-4cf8-a700-33bba4fa9893" (UID: "1fd4d728-3c76-4cf8-a700-33bba4fa9893"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:01.463087 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.463063 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd4d728-3c76-4cf8-a700-33bba4fa9893-kube-api-access-rwfnx" pod "1fd4d728-3c76-4cf8-a700-33bba4fa9893" (UID: "1fd4d728-3c76-4cf8-a700-33bba4fa9893"). InnerVolumeSpecName "kube-api-access-rwfnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:01.463087 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.463064 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-tls-certs" pod "1fd4d728-3c76-4cf8-a700-33bba4fa9893" (UID: "1fd4d728-3c76-4cf8-a700-33bba4fa9893"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:01.561546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.561522 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:01.561546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.561546 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:01.561724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.561556 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:01.561724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.561566 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:01.561724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.561574 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwfnx\" (UniqueName: \"kubernetes.io/projected/1fd4d728-3c76-4cf8-a700-33bba4fa9893-kube-api-access-rwfnx\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:01.561724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.561582 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd4d728-3c76-4cf8-a700-33bba4fa9893-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:01.589157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.589134 5014 generic.go:356] "Generic (PLEG): container finished" podID="1fd4d728-3c76-4cf8-a700-33bba4fa9893" containerID="0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da" exitCode=0 Oct 30 18:47:01.589250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.589192 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.589250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.589210 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1fd4d728-3c76-4cf8-a700-33bba4fa9893","Type":"ContainerDied","Data":"0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da"} Oct 30 18:47:01.589328 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.589279 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1fd4d728-3c76-4cf8-a700-33bba4fa9893","Type":"ContainerDied","Data":"c02e6b1c9850ffb6b52208331ea44bb081700b1986ef9ed745cc78fb47c9a26f"} Oct 30 18:47:01.589328 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.589302 5014 scope.go:122] "RemoveContainer" containerID="0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da" Oct 30 18:47:01.600886 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.600866 5014 scope.go:122] "RemoveContainer" containerID="0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da" Oct 30 18:47:01.601325 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:01.601304 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da\": container with ID starting with 0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da not found: ID does not exist" containerID="0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da" Oct 30 18:47:01.601425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.601333 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da"} err="failed to get container status \"0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da\": rpc error: code = NotFound desc = could not find container \"0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da\": container with ID starting with 0b8a20a79a10e18e535519653d6c07726a968eb56aacd00c6294d1ab6ed1b2da not found: ID does not exist" Oct 30 18:47:01.638712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.638688 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:01.644359 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.644341 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:01.965291 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.965210 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:01.993296 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:01.993270 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:02.066610 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:02.066569 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:02.066731 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:02.066709 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:02.080639 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:02.080617 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:02.181059 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:02.181032 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:02.207727 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:02.207705 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:02.271194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:02.271145 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:02.598323 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:02.598293 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:47:02.615956 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:47:02.615931 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01ff8e3_24cf_4745_adbf_8fc07c3725b4.slice/crio-4b3d215bf967788403e1c917122cc3718d7292cc796b9974ff1517f284f4e306 WatchSource:0}: Error finding container 4b3d215bf967788403e1c917122cc3718d7292cc796b9974ff1517f284f4e306: Status 404 returned error can't find the container with id 4b3d215bf967788403e1c917122cc3718d7292cc796b9974ff1517f284f4e306 Oct 30 18:47:03.142631 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:03.142580 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd4d728-3c76-4cf8-a700-33bba4fa9893" path="/var/lib/kubelet/pods/1fd4d728-3c76-4cf8-a700-33bba4fa9893/volumes" Oct 30 18:47:03.603042 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:03.603008 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"b01ff8e3-24cf-4745-adbf-8fc07c3725b4","Type":"ContainerStarted","Data":"79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5"} Oct 30 18:47:03.603499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:03.603067 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"b01ff8e3-24cf-4745-adbf-8fc07c3725b4","Type":"ContainerStarted","Data":"23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290"} Oct 30 18:47:03.603499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:03.603086 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"b01ff8e3-24cf-4745-adbf-8fc07c3725b4","Type":"ContainerStarted","Data":"f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85"} Oct 30 18:47:03.603499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:03.603191 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:03.603499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:03.603244 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"b01ff8e3-24cf-4745-adbf-8fc07c3725b4","Type":"ContainerStarted","Data":"4b3d215bf967788403e1c917122cc3718d7292cc796b9974ff1517f284f4e306"} Oct 30 18:47:03.603499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:03.603283 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:03.603499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:03.603310 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:03.605896 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:03.605879 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:03.611879 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:03.611862 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:03.679748 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:03.679698 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=4.6796826540000005 podStartE2EDuration="4.679682654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:03.667057035 +0000 UTC m=+3036.655397522" watchObservedRunningTime="2026-10-30 18:47:03.679682654 +0000 UTC m=+3036.668023143" Oct 30 18:47:13.333931 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:13.333886 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:47:16.582697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:16.582647 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:47:16.583061 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:16.582910 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo" containerID="cri-o://f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85" gracePeriod=30 Oct 30 18:47:16.583319 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:16.583294 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo-gateway-opa" containerID="cri-o://79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5" gracePeriod=30 Oct 30 18:47:16.583428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:16.583412 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo-gateway" containerID="cri-o://23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290" gracePeriod=30 Oct 30 18:47:17.672161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.672134 5014 generic.go:356] "Generic (PLEG): container finished" podID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerID="79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5" exitCode=0 Oct 30 18:47:17.672161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.672156 5014 generic.go:356] "Generic (PLEG): container finished" podID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerID="f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85" exitCode=0 Oct 30 18:47:17.672511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.672208 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"b01ff8e3-24cf-4745-adbf-8fc07c3725b4","Type":"ContainerDied","Data":"79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5"} Oct 30 18:47:17.672511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.672257 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"b01ff8e3-24cf-4745-adbf-8fc07c3725b4","Type":"ContainerDied","Data":"f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85"} Oct 30 18:47:17.831094 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.831071 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:17.994645 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.994559 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-kube-api-access-84glx\" (UniqueName: \"kubernetes.io/projected/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-kube-api-access-84glx\") pod \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " Oct 30 18:47:17.994645 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.994610 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-tenants\") pod \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " Oct 30 18:47:17.994862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.994654 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-rbac\") pod \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " Oct 30 18:47:17.994862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.994687 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " Oct 30 18:47:17.994862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.994706 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-ca-bundle\") pod \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " Oct 30 18:47:17.994862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.994749 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-mtls\") pod \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " Oct 30 18:47:17.995101 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.994860 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " Oct 30 18:47:17.995101 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.994914 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " Oct 30 18:47:17.995101 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.994948 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-conf\") pod \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\" (UID: \"b01ff8e3-24cf-4745-adbf-8fc07c3725b4\") " Oct 30 18:47:17.995101 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.995075 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-rbac" pod "b01ff8e3-24cf-4745-adbf-8fc07c3725b4" (UID: "b01ff8e3-24cf-4745-adbf-8fc07c3725b4"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:17.995325 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.995189 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-ca-bundle" pod "b01ff8e3-24cf-4745-adbf-8fc07c3725b4" (UID: "b01ff8e3-24cf-4745-adbf-8fc07c3725b4"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:17.995479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.995368 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-conf" pod "b01ff8e3-24cf-4745-adbf-8fc07c3725b4" (UID: "b01ff8e3-24cf-4745-adbf-8fc07c3725b4"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:17.995635 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.995563 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:17.995704 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.995561 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-serving-cabundle" pod "b01ff8e3-24cf-4745-adbf-8fc07c3725b4" (UID: "b01ff8e3-24cf-4745-adbf-8fc07c3725b4"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:17.997183 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.997145 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-kube-api-access-84glx" pod "b01ff8e3-24cf-4745-adbf-8fc07c3725b4" (UID: "b01ff8e3-24cf-4745-adbf-8fc07c3725b4"). InnerVolumeSpecName "kube-api-access-84glx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:17.997319 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.997297 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-tenants" pod "b01ff8e3-24cf-4745-adbf-8fc07c3725b4" (UID: "b01ff8e3-24cf-4745-adbf-8fc07c3725b4"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:17.997620 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.997584 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "b01ff8e3-24cf-4745-adbf-8fc07c3725b4" (UID: "b01ff8e3-24cf-4745-adbf-8fc07c3725b4"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:17.997765 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.997744 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-mtls" pod "b01ff8e3-24cf-4745-adbf-8fc07c3725b4" (UID: "b01ff8e3-24cf-4745-adbf-8fc07c3725b4"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:17.998359 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:17.998342 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "b01ff8e3-24cf-4745-adbf-8fc07c3725b4" (UID: "b01ff8e3-24cf-4745-adbf-8fc07c3725b4"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:47:18.096937 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.096916 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:18.096937 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.096938 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:18.097105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.096948 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:18.097105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.096970 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:47:18.097105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.096980 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:18.097105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.096990 5014 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-tempo-conf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:18.097105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.096999 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84glx\" (UniqueName: \"kubernetes.io/projected/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-kube-api-access-84glx\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:18.097105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.097007 5014 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/b01ff8e3-24cf-4745-adbf-8fc07c3725b4-gateway-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:18.105991 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.105976 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:47:18.198323 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.198298 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-36935df131a4fd8016f25ef785835777a9b31b51817cbdc6cffb4b5510e5f60e\"" Oct 30 18:47:18.679425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.679391 5014 generic.go:356] "Generic (PLEG): container finished" podID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerID="23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290" exitCode=0 Oct 30 18:47:18.679873 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.679446 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:47:18.679873 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.679475 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"b01ff8e3-24cf-4745-adbf-8fc07c3725b4","Type":"ContainerDied","Data":"23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290"} Oct 30 18:47:18.679873 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.679549 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"b01ff8e3-24cf-4745-adbf-8fc07c3725b4","Type":"ContainerDied","Data":"4b3d215bf967788403e1c917122cc3718d7292cc796b9974ff1517f284f4e306"} Oct 30 18:47:18.679873 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.679585 5014 scope.go:122] "RemoveContainer" containerID="79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5" Oct 30 18:47:18.689483 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.689467 5014 scope.go:122] "RemoveContainer" containerID="23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290" Oct 30 18:47:18.697738 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.697533 5014 scope.go:122] "RemoveContainer" containerID="f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85" Oct 30 18:47:18.705342 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.705325 5014 scope.go:122] "RemoveContainer" containerID="79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5" Oct 30 18:47:18.705651 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:18.705624 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5\": container with ID starting with 79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5 not found: ID does not exist" containerID="79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5" Oct 30 18:47:18.705758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.705659 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5"} err="failed to get container status \"79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5\": rpc error: code = NotFound desc = could not find container \"79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5\": container with ID starting with 79cde473e0d15a8771c31d65828f66b433beca009fba4985afbf164f25deb9a5 not found: ID does not exist" Oct 30 18:47:18.705758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.705678 5014 scope.go:122] "RemoveContainer" containerID="23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290" Oct 30 18:47:18.705904 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:18.705888 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290\": container with ID starting with 23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290 not found: ID does not exist" containerID="23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290" Oct 30 18:47:18.705946 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.705908 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290"} err="failed to get container status \"23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290\": rpc error: code = NotFound desc = could not find container \"23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290\": container with ID starting with 23de993aca2c389598f843914606db4f2204bb92c0519dd311f071d9a1734290 not found: ID does not exist" Oct 30 18:47:18.705946 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.705922 5014 scope.go:122] "RemoveContainer" containerID="f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85" Oct 30 18:47:18.706168 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:18.706153 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85\": container with ID starting with f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85 not found: ID does not exist" containerID="f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85" Oct 30 18:47:18.706210 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.706171 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85"} err="failed to get container status \"f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85\": rpc error: code = NotFound desc = could not find container \"f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85\": container with ID starting with f52c4eb54659150069cb66982f345424a2589fe35d666a3121cd7260327c6d85 not found: ID does not exist" Oct 30 18:47:18.742370 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.742345 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:47:18.753973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:18.753951 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:47:19.141498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:19.141472 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" path="/var/lib/kubelet/pods/b01ff8e3-24cf-4745-adbf-8fc07c3725b4/volumes" Oct 30 18:47:20.840727 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.840696 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:20.841132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841051 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo-gateway" Oct 30 18:47:20.841132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841064 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo-gateway" Oct 30 18:47:20.841132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841086 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo-gateway-opa" Oct 30 18:47:20.841132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841095 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo-gateway-opa" Oct 30 18:47:20.841132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841110 5014 cpu_manager.go:414] "containerMap: removing container" podUID="1fd4d728-3c76-4cf8-a700-33bba4fa9893" containerName="otc-container" Oct 30 18:47:20.841132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841115 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd4d728-3c76-4cf8-a700-33bba4fa9893" containerName="otc-container" Oct 30 18:47:20.841132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841130 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo" Oct 30 18:47:20.841416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841137 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo" Oct 30 18:47:20.841416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841229 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo-gateway-opa" Oct 30 18:47:20.841416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841245 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd4d728-3c76-4cf8-a700-33bba4fa9893" containerName="otc-container" Oct 30 18:47:20.841416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841257 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo" Oct 30 18:47:20.841416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.841269 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01ff8e3-24cf-4745-adbf-8fc07c3725b4" containerName="tempo-gateway" Oct 30 18:47:20.846022 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.846003 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:20.872459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.872439 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:47:20.872573 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.872456 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:47:20.872573 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.872491 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:47:20.872573 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.872501 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:47:20.872573 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.872512 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:47:20.883680 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.883662 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsglb\"" Oct 30 18:47:20.885351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:20.885290 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:21.017852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.017820 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.017955 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.017879 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.017998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.017953 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.018033 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.017996 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt6dm\" (UniqueName: \"kubernetes.io/projected/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-kube-api-access-rt6dm\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.018033 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.018024 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.018107 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.018045 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.118546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.118485 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.118546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.118533 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt6dm\" (UniqueName: \"kubernetes.io/projected/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-kube-api-access-rt6dm\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.118707 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.118556 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.118707 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.118660 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.118778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.118708 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.118821 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.118780 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.119717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.119690 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.121282 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.121260 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.121393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.121321 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.121393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.121366 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.121393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.121367 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.141503 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.141482 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt6dm\" (UniqueName: \"kubernetes.io/projected/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-kube-api-access-rt6dm\") pod \"data-science-collector-collector-0\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.159229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.159209 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:21.455222 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.455150 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:21.459851 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:47:21.459826 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b3fef0_8e91_4d98_bf4e_4c8e0574372e.slice/crio-b795dfce01878e2fe20e3db5fd4c10acae8820d988f3c654eeb6464d2c9b9e41 WatchSource:0}: Error finding container b795dfce01878e2fe20e3db5fd4c10acae8820d988f3c654eeb6464d2c9b9e41: Status 404 returned error can't find the container with id b795dfce01878e2fe20e3db5fd4c10acae8820d988f3c654eeb6464d2c9b9e41 Oct 30 18:47:21.461834 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.461803 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:21.461932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.461882 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:21.697080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.697043 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"39b3fef0-8e91-4d98-bf4e-4c8e0574372e","Type":"ContainerStarted","Data":"6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8"} Oct 30 18:47:21.697222 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.697103 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"39b3fef0-8e91-4d98-bf4e-4c8e0574372e","Type":"ContainerStarted","Data":"b795dfce01878e2fe20e3db5fd4c10acae8820d988f3c654eeb6464d2c9b9e41"} Oct 30 18:47:21.740348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:21.740220 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.740207338 podStartE2EDuration="1.740207338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:21.738802767 +0000 UTC m=+3054.727143267" watchObservedRunningTime="2026-10-30 18:47:21.740207338 +0000 UTC m=+3054.728547898" Oct 30 18:47:24.544282 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.544241 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2"] Oct 30 18:47:24.544913 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.544540 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" podUID="e7797026-78dd-49a6-823d-e904cba88ab1" containerName="kube-rbac-proxy" containerID="cri-o://90d634b1fb404380fe36a4f0be4123b49c8cfa458514434a3d98b469363cfd04" gracePeriod=30 Oct 30 18:47:24.714710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.714673 5014 generic.go:356] "Generic (PLEG): container finished" podID="e7797026-78dd-49a6-823d-e904cba88ab1" containerID="90d634b1fb404380fe36a4f0be4123b49c8cfa458514434a3d98b469363cfd04" exitCode=0 Oct 30 18:47:24.714862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.714712 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" event={"ID":"e7797026-78dd-49a6-823d-e904cba88ab1","Type":"ContainerDied","Data":"90d634b1fb404380fe36a4f0be4123b49c8cfa458514434a3d98b469363cfd04"} Oct 30 18:47:24.806117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.806096 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:47:24.882780 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.882751 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6"] Oct 30 18:47:24.882966 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.882927 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" podUID="6aadb74c-066c-483b-9cea-f9114a5a596d" containerName="thanos-querier" containerID="cri-o://83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050" gracePeriod=30 Oct 30 18:47:24.950827 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.950796 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-ca\") pod \"e7797026-78dd-49a6-823d-e904cba88ab1\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " Oct 30 18:47:24.950978 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.950876 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-kube-rbac-proxy-config\") pod \"e7797026-78dd-49a6-823d-e904cba88ab1\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " Oct 30 18:47:24.950978 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.950943 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7797026-78dd-49a6-823d-e904cba88ab1-kube-api-access-g757p\" (UniqueName: \"kubernetes.io/projected/e7797026-78dd-49a6-823d-e904cba88ab1-kube-api-access-g757p\") pod \"e7797026-78dd-49a6-823d-e904cba88ab1\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " Oct 30 18:47:24.951106 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.950979 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-client-cert\") pod \"e7797026-78dd-49a6-823d-e904cba88ab1\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " Oct 30 18:47:24.951106 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.951021 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-tls\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-tls\") pod \"e7797026-78dd-49a6-823d-e904cba88ab1\" (UID: \"e7797026-78dd-49a6-823d-e904cba88ab1\") " Oct 30 18:47:24.951219 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.951135 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-ca" pod "e7797026-78dd-49a6-823d-e904cba88ab1" (UID: "e7797026-78dd-49a6-823d-e904cba88ab1"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:24.951352 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.951331 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:24.953453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.953420 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7797026-78dd-49a6-823d-e904cba88ab1-kube-api-access-g757p" pod "e7797026-78dd-49a6-823d-e904cba88ab1" (UID: "e7797026-78dd-49a6-823d-e904cba88ab1"). InnerVolumeSpecName "kube-api-access-g757p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:24.953572 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.953452 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-kube-rbac-proxy-config" pod "e7797026-78dd-49a6-823d-e904cba88ab1" (UID: "e7797026-78dd-49a6-823d-e904cba88ab1"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:24.954049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.954028 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-tls" pod "e7797026-78dd-49a6-823d-e904cba88ab1" (UID: "e7797026-78dd-49a6-823d-e904cba88ab1"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:24.954478 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:24.954463 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-client-cert" pod "e7797026-78dd-49a6-823d-e904cba88ab1" (UID: "e7797026-78dd-49a6-823d-e904cba88ab1"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.053270 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.053227 5014 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-kube-rbac-proxy-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.053270 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.053274 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g757p\" (UniqueName: \"kubernetes.io/projected/e7797026-78dd-49a6-823d-e904cba88ab1-kube-api-access-g757p\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.053537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.053291 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-prometheus-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.053537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.053321 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e7797026-78dd-49a6-823d-e904cba88ab1-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.067307 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.067275 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:47:25.067582 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.067559 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="prometheus" containerID="cri-o://b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc" gracePeriod=600 Oct 30 18:47:25.067941 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.067890 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="config-reloader" containerID="cri-o://5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432" gracePeriod=600 Oct 30 18:47:25.070209 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.067827 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="thanos-sidecar" containerID="cri-o://a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884" gracePeriod=600 Oct 30 18:47:25.070539 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.070519 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:47:25.071251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.071061 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="39107235-3fb6-49af-b295-9be59c67c288" containerName="alertmanager" containerID="cri-o://fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19" gracePeriod=120 Oct 30 18:47:25.071251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.071141 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="39107235-3fb6-49af-b295-9be59c67c288" containerName="config-reloader" containerID="cri-o://df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3" gracePeriod=120 Oct 30 18:47:25.126492 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.126461 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="prometheus" probeResult="failure" output="Get \"https://10.132.0.111:9090/-/ready\": dial tcp 10.132.0.111:9090: connect: connection refused" Oct 30 18:47:25.283756 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.283733 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" Oct 30 18:47:25.356052 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.356029 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aadb74c-066c-483b-9cea-f9114a5a596d-kube-api-access-nzq7p\" (UniqueName: \"kubernetes.io/projected/6aadb74c-066c-483b-9cea-f9114a5a596d-kube-api-access-nzq7p\") pod \"6aadb74c-066c-483b-9cea-f9114a5a596d\" (UID: \"6aadb74c-066c-483b-9cea-f9114a5a596d\") " Oct 30 18:47:25.358664 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.358641 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aadb74c-066c-483b-9cea-f9114a5a596d-kube-api-access-nzq7p" pod "6aadb74c-066c-483b-9cea-f9114a5a596d" (UID: "6aadb74c-066c-483b-9cea-f9114a5a596d"). InnerVolumeSpecName "kube-api-access-nzq7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:25.413169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.413147 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:25.457027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457007 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-thanos-prometheus-http-client-file\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457040 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457067 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457095 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config-out\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457117 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457333 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457145 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-secret-prometheus-web-tls-ca\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457333 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457167 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457333 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457197 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457333 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457227 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457333 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457259 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-kube-api-access-8b6nr\" (UniqueName: \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-kube-api-access-8b6nr\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457589 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457367 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457589 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457445 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-tls-assets\" (UniqueName: \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-tls-assets\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457589 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457481 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\" (UID: \"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c\") " Oct 30 18:47:25.457938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.457789 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzq7p\" (UniqueName: \"kubernetes.io/projected/6aadb74c-066c-483b-9cea-f9114a5a596d-kube-api-access-nzq7p\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.458072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.458016 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-1" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.458634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.458376 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-0" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.458634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.458384 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-2" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.459939 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.459908 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config-out" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:47:25.460046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.459927 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-thanos-prometheus-http-client-file" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.460161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.460133 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.460337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.460293 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.460523 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.460502 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-secret-prometheus-web-tls-ca" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.460984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.460963 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.461840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.461796 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-kube-api-access-8b6nr" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "kube-api-access-8b6nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:25.462291 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.462270 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-tls-assets" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:25.463665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.463643 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:47:25.476559 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.476529 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config" pod "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" (UID: "cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.558421 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558395 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558421 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558420 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558431 5014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558443 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558452 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558461 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558469 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558479 5014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558487 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558495 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558504 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558513 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6nr\" (UniqueName: \"kubernetes.io/projected/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c-kube-api-access-8b6nr\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.558820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.558537 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:47:25.570709 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.570692 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:47:25.635933 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.635875 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:25.636104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.636081 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="39b3fef0-8e91-4d98-bf4e-4c8e0574372e" containerName="otc-container" containerID="cri-o://6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8" gracePeriod=30 Oct 30 18:47:25.659758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.659734 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-3ece98a829a66f7cef6e6b4403002493a48d56f442e1673d82fff8fda237a009\"" Oct 30 18:47:25.726356 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.726320 5014 generic.go:356] "Generic (PLEG): container finished" podID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerID="a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884" exitCode=0 Oct 30 18:47:25.726356 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.726352 5014 generic.go:356] "Generic (PLEG): container finished" podID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerID="5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432" exitCode=0 Oct 30 18:47:25.726609 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.726365 5014 generic.go:356] "Generic (PLEG): container finished" podID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerID="b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc" exitCode=0 Oct 30 18:47:25.726609 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.726392 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:25.726609 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.726392 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c","Type":"ContainerDied","Data":"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884"} Oct 30 18:47:25.726609 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.726467 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c","Type":"ContainerDied","Data":"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432"} Oct 30 18:47:25.726609 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.726505 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c","Type":"ContainerDied","Data":"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc"} Oct 30 18:47:25.726609 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.726543 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c","Type":"ContainerDied","Data":"3d526c86d661743e31f4a61add01a6acb9478490c834aa8abecb3a5fab8df03d"} Oct 30 18:47:25.726941 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.726587 5014 scope.go:122] "RemoveContainer" containerID="a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884" Oct 30 18:47:25.731505 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.729156 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" Oct 30 18:47:25.731505 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.729242 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2" event={"ID":"e7797026-78dd-49a6-823d-e904cba88ab1","Type":"ContainerDied","Data":"cff53a8ea3aa183dab0e698aef43f95b0c7ff5853c8748cfa84666a29c11bc61"} Oct 30 18:47:25.731505 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.731478 5014 generic.go:356] "Generic (PLEG): container finished" podID="6aadb74c-066c-483b-9cea-f9114a5a596d" containerID="83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050" exitCode=0 Oct 30 18:47:25.731800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.731571 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" event={"ID":"6aadb74c-066c-483b-9cea-f9114a5a596d","Type":"ContainerDied","Data":"83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050"} Oct 30 18:47:25.731800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.731617 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" event={"ID":"6aadb74c-066c-483b-9cea-f9114a5a596d","Type":"ContainerDied","Data":"1636a8e3dc7eacb25e699e6592ca7690fb870ddf0ff66f0c7dbf07b651a2fbfc"} Oct 30 18:47:25.731800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.731682 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6" Oct 30 18:47:25.735013 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.734949 5014 generic.go:356] "Generic (PLEG): container finished" podID="39107235-3fb6-49af-b295-9be59c67c288" containerID="df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3" exitCode=0 Oct 30 18:47:25.735128 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.735019 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39107235-3fb6-49af-b295-9be59c67c288","Type":"ContainerDied","Data":"df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3"} Oct 30 18:47:25.803226 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.799969 5014 scope.go:122] "RemoveContainer" containerID="5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432" Oct 30 18:47:25.814642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.814623 5014 scope.go:122] "RemoveContainer" containerID="b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc" Oct 30 18:47:25.825719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.825695 5014 scope.go:122] "RemoveContainer" containerID="03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9" Oct 30 18:47:25.835675 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.833697 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:47:25.835675 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.835281 5014 scope.go:122] "RemoveContainer" containerID="a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.839113 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:25.839234 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884\": container with ID starting with a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884 not found: ID does not exist" containerID="a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.839262 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884"} err="failed to get container status \"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884\": rpc error: code = NotFound desc = could not find container \"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884\": container with ID starting with a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884 not found: ID does not exist" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.839288 5014 scope.go:122] "RemoveContainer" containerID="5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:25.839620 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432\": container with ID starting with 5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432 not found: ID does not exist" containerID="5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.839647 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432"} err="failed to get container status \"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432\": rpc error: code = NotFound desc = could not find container \"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432\": container with ID starting with 5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432 not found: ID does not exist" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.839663 5014 scope.go:122] "RemoveContainer" containerID="b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:25.839932 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc\": container with ID starting with b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc not found: ID does not exist" containerID="b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.839956 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc"} err="failed to get container status \"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc\": rpc error: code = NotFound desc = could not find container \"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc\": container with ID starting with b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc not found: ID does not exist" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.839971 5014 scope.go:122] "RemoveContainer" containerID="03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:25.840277 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9\": container with ID starting with 03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9 not found: ID does not exist" containerID="03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.840314 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9"} err="failed to get container status \"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9\": rpc error: code = NotFound desc = could not find container \"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9\": container with ID starting with 03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9 not found: ID does not exist" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.840348 5014 scope.go:122] "RemoveContainer" containerID="a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.840658 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884"} err="failed to get container status \"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884\": rpc error: code = NotFound desc = could not find container \"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884\": container with ID starting with a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884 not found: ID does not exist" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.840676 5014 scope.go:122] "RemoveContainer" containerID="5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.840980 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432"} err="failed to get container status \"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432\": rpc error: code = NotFound desc = could not find container \"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432\": container with ID starting with 5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432 not found: ID does not exist" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.840998 5014 scope.go:122] "RemoveContainer" containerID="b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.841227 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc"} err="failed to get container status \"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc\": rpc error: code = NotFound desc = could not find container \"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc\": container with ID starting with b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc not found: ID does not exist" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.841247 5014 scope.go:122] "RemoveContainer" containerID="03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.841477 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9"} err="failed to get container status \"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9\": rpc error: code = NotFound desc = could not find container \"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9\": container with ID starting with 03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9 not found: ID does not exist" Oct 30 18:47:25.841616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.841503 5014 scope.go:122] "RemoveContainer" containerID="a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884" Oct 30 18:47:25.842437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.841814 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884"} err="failed to get container status \"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884\": rpc error: code = NotFound desc = could not find container \"a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884\": container with ID starting with a8cfe3d8df94fb3e0f423b51a809896c674c042760bc465602ed270399c65884 not found: ID does not exist" Oct 30 18:47:25.842437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.841840 5014 scope.go:122] "RemoveContainer" containerID="5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432" Oct 30 18:47:25.842437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.842084 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432"} err="failed to get container status \"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432\": rpc error: code = NotFound desc = could not find container \"5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432\": container with ID starting with 5aedaeda02d13f8980315fd1a84eb132c4e61510e0881c05c7287b2f89721432 not found: ID does not exist" Oct 30 18:47:25.842437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.842099 5014 scope.go:122] "RemoveContainer" containerID="b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc" Oct 30 18:47:25.842437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.842361 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc"} err="failed to get container status \"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc\": rpc error: code = NotFound desc = could not find container \"b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc\": container with ID starting with b095127732cdd017b08407d50697cb4bdd318079c143ca58ac431a7efcd8f0cc not found: ID does not exist" Oct 30 18:47:25.842437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.842384 5014 scope.go:122] "RemoveContainer" containerID="03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9" Oct 30 18:47:25.843461 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.842655 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9"} err="failed to get container status \"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9\": rpc error: code = NotFound desc = could not find container \"03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9\": container with ID starting with 03c7a8405ff5ad714763df3c77a3e7a6b9e8841d13c3b15b06b19768d86372f9 not found: ID does not exist" Oct 30 18:47:25.843461 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.842681 5014 scope.go:122] "RemoveContainer" containerID="90d634b1fb404380fe36a4f0be4123b49c8cfa458514434a3d98b469363cfd04" Oct 30 18:47:25.855041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.855029 5014 scope.go:122] "RemoveContainer" containerID="83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050" Oct 30 18:47:25.867112 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.867091 5014 scope.go:122] "RemoveContainer" containerID="83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050" Oct 30 18:47:25.867436 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:25.867398 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050\": container with ID starting with 83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050 not found: ID does not exist" containerID="83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050" Oct 30 18:47:25.867534 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.867426 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050"} err="failed to get container status \"83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050\": rpc error: code = NotFound desc = could not find container \"83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050\": container with ID starting with 83e4e25c09a542effdbcb10a1d060622ce64587d0fb81790499ed6e1c85e1050 not found: ID does not exist" Oct 30 18:47:25.870993 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.870972 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6"] Oct 30 18:47:25.874887 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.874642 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ppdv6"] Oct 30 18:47:25.910424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.910389 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2"] Oct 30 18:47:25.917677 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.917652 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hm7r2"] Oct 30 18:47:25.949157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.949134 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:25.962227 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.962208 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otlp-tls-certs\") pod \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " Oct 30 18:47:25.962344 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.962252 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-data-science-collector-ta-client-cert\") pod \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " Oct 30 18:47:25.962418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.962367 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-monitoring-tls-certs\") pod \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " Oct 30 18:47:25.962477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.962424 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-tls-certs\") pod \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " Oct 30 18:47:25.962477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.962459 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otc-internal\") pod \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " Oct 30 18:47:25.962622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.962493 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-kube-api-access-rt6dm\" (UniqueName: \"kubernetes.io/projected/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-kube-api-access-rt6dm\") pod \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\" (UID: \"39b3fef0-8e91-4d98-bf4e-4c8e0574372e\") " Oct 30 18:47:25.963570 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.963526 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otc-internal" pod "39b3fef0-8e91-4d98-bf4e-4c8e0574372e" (UID: "39b3fef0-8e91-4d98-bf4e-4c8e0574372e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.965087 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.965062 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-monitoring-tls-certs" pod "39b3fef0-8e91-4d98-bf4e-4c8e0574372e" (UID: "39b3fef0-8e91-4d98-bf4e-4c8e0574372e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.965199 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.965146 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-data-science-collector-ta-client-cert" pod "39b3fef0-8e91-4d98-bf4e-4c8e0574372e" (UID: "39b3fef0-8e91-4d98-bf4e-4c8e0574372e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.965323 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.965303 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otlp-tls-certs" pod "39b3fef0-8e91-4d98-bf4e-4c8e0574372e" (UID: "39b3fef0-8e91-4d98-bf4e-4c8e0574372e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.965384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.965346 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-tls-certs" pod "39b3fef0-8e91-4d98-bf4e-4c8e0574372e" (UID: "39b3fef0-8e91-4d98-bf4e-4c8e0574372e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.965676 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.965660 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-kube-api-access-rt6dm" pod "39b3fef0-8e91-4d98-bf4e-4c8e0574372e" (UID: "39b3fef0-8e91-4d98-bf4e-4c8e0574372e"). InnerVolumeSpecName "kube-api-access-rt6dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:25.966298 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:25.966278 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:26.063954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.063918 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-web-config\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-web-config\") pod \"39107235-3fb6-49af-b295-9be59c67c288\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " Oct 30 18:47:26.063954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.063958 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-config-out\" (UniqueName: \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-config-out\") pod \"39107235-3fb6-49af-b295-9be59c67c288\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " Oct 30 18:47:26.064194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.063989 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-kube-api-access-dwzj9\" (UniqueName: \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-kube-api-access-dwzj9\") pod \"39107235-3fb6-49af-b295-9be59c67c288\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " Oct 30 18:47:26.064194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064041 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-alertmanager-data-science-monitoringstack-db\") pod \"39107235-3fb6-49af-b295-9be59c67c288\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " Oct 30 18:47:26.064297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064227 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-cluster-tls-config\") pod \"39107235-3fb6-49af-b295-9be59c67c288\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " Oct 30 18:47:26.064349 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064333 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-config-volume\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-config-volume\") pod \"39107235-3fb6-49af-b295-9be59c67c288\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " Oct 30 18:47:26.064389 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064374 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-tls-assets\" (UniqueName: \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-tls-assets\") pod \"39107235-3fb6-49af-b295-9be59c67c288\" (UID: \"39107235-3fb6-49af-b295-9be59c67c288\") " Oct 30 18:47:26.064439 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064399 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-alertmanager-data-science-monitoringstack-db" pod "39107235-3fb6-49af-b295-9be59c67c288" (UID: "39107235-3fb6-49af-b295-9be59c67c288"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:47:26.064853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064657 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.064853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064678 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.064853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064694 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.064853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064707 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt6dm\" (UniqueName: \"kubernetes.io/projected/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-kube-api-access-rt6dm\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.064853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064720 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.064853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064734 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/39b3fef0-8e91-4d98-bf4e-4c8e0574372e-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.064853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.064748 5014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.066645 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.066588 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-config-volume" pod "39107235-3fb6-49af-b295-9be59c67c288" (UID: "39107235-3fb6-49af-b295-9be59c67c288"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:26.066927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.066907 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-kube-api-access-dwzj9" pod "39107235-3fb6-49af-b295-9be59c67c288" (UID: "39107235-3fb6-49af-b295-9be59c67c288"). InnerVolumeSpecName "kube-api-access-dwzj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:26.067224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.067197 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-tls-assets" pod "39107235-3fb6-49af-b295-9be59c67c288" (UID: "39107235-3fb6-49af-b295-9be59c67c288"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:26.067296 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.067254 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-config-out" pod "39107235-3fb6-49af-b295-9be59c67c288" (UID: "39107235-3fb6-49af-b295-9be59c67c288"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:47:26.072304 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.072282 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-cluster-tls-config" pod "39107235-3fb6-49af-b295-9be59c67c288" (UID: "39107235-3fb6-49af-b295-9be59c67c288"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:26.079537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.079516 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-web-config" pod "39107235-3fb6-49af-b295-9be59c67c288" (UID: "39107235-3fb6-49af-b295-9be59c67c288"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:26.166089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.166031 5014 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-cluster-tls-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.166089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.166055 5014 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-config-volume\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.166089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.166068 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.166089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.166081 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39107235-3fb6-49af-b295-9be59c67c288-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.166272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.166093 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39107235-3fb6-49af-b295-9be59c67c288-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.166272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.166106 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwzj9\" (UniqueName: \"kubernetes.io/projected/39107235-3fb6-49af-b295-9be59c67c288-kube-api-access-dwzj9\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.741879 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.741847 5014 generic.go:356] "Generic (PLEG): container finished" podID="39b3fef0-8e91-4d98-bf4e-4c8e0574372e" containerID="6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8" exitCode=0 Oct 30 18:47:26.742340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.741890 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:26.742340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.741918 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"39b3fef0-8e91-4d98-bf4e-4c8e0574372e","Type":"ContainerDied","Data":"6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8"} Oct 30 18:47:26.742340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.741962 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"39b3fef0-8e91-4d98-bf4e-4c8e0574372e","Type":"ContainerDied","Data":"b795dfce01878e2fe20e3db5fd4c10acae8820d988f3c654eeb6464d2c9b9e41"} Oct 30 18:47:26.742340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.741973 5014 scope.go:122] "RemoveContainer" containerID="6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8" Oct 30 18:47:26.746155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.746131 5014 generic.go:356] "Generic (PLEG): container finished" podID="39107235-3fb6-49af-b295-9be59c67c288" containerID="fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19" exitCode=0 Oct 30 18:47:26.746255 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.746163 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39107235-3fb6-49af-b295-9be59c67c288","Type":"ContainerDied","Data":"fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19"} Oct 30 18:47:26.746255 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.746199 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:26.746255 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.746212 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39107235-3fb6-49af-b295-9be59c67c288","Type":"ContainerDied","Data":"105e9772e4840a15775360df7378b3bd98b2bf8073c92a4781758870d5a0194c"} Oct 30 18:47:26.753022 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.753003 5014 scope.go:122] "RemoveContainer" containerID="6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8" Oct 30 18:47:26.753277 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:26.753257 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8\": container with ID starting with 6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8 not found: ID does not exist" containerID="6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8" Oct 30 18:47:26.753350 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.753288 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8"} err="failed to get container status \"6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8\": rpc error: code = NotFound desc = could not find container \"6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8\": container with ID starting with 6bd988baf70af8e7df77330f800ed542dfc156492e369c2d1135c3f6dc295be8 not found: ID does not exist" Oct 30 18:47:26.753350 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.753311 5014 scope.go:122] "RemoveContainer" containerID="df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3" Oct 30 18:47:26.761387 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.761370 5014 scope.go:122] "RemoveContainer" containerID="fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19" Oct 30 18:47:26.768756 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.768742 5014 scope.go:122] "RemoveContainer" containerID="a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb" Oct 30 18:47:26.775907 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.775892 5014 scope.go:122] "RemoveContainer" containerID="df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3" Oct 30 18:47:26.776187 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:26.776171 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3\": container with ID starting with df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3 not found: ID does not exist" containerID="df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3" Oct 30 18:47:26.776236 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.776193 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3"} err="failed to get container status \"df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3\": rpc error: code = NotFound desc = could not find container \"df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3\": container with ID starting with df060436920a25fdc6f98bb5afd88c3422247112ea461b3597353f8edc9785d3 not found: ID does not exist" Oct 30 18:47:26.776236 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.776208 5014 scope.go:122] "RemoveContainer" containerID="fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19" Oct 30 18:47:26.776462 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:26.776443 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19\": container with ID starting with fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19 not found: ID does not exist" containerID="fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19" Oct 30 18:47:26.776530 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.776472 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19"} err="failed to get container status \"fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19\": rpc error: code = NotFound desc = could not find container \"fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19\": container with ID starting with fd0926d438530d93248fac2760262d5d6df96ca9d4ed6f47fba29940fb343d19 not found: ID does not exist" Oct 30 18:47:26.776530 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.776501 5014 scope.go:122] "RemoveContainer" containerID="a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb" Oct 30 18:47:26.776761 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:26.776747 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb\": container with ID starting with a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb not found: ID does not exist" containerID="a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb" Oct 30 18:47:26.776806 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.776765 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb"} err="failed to get container status \"a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb\": rpc error: code = NotFound desc = could not find container \"a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb\": container with ID starting with a013996edfa026656604fc64fb8649a1449e4913b3f2752b5199b3dc9b6205eb not found: ID does not exist" Oct 30 18:47:26.797202 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.797181 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:26.799332 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.799312 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:26.853154 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.853102 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:47:26.859567 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:26.859547 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:47:27.140534 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:27.140478 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39107235-3fb6-49af-b295-9be59c67c288" path="/var/lib/kubelet/pods/39107235-3fb6-49af-b295-9be59c67c288/volumes" Oct 30 18:47:27.140971 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:27.140957 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b3fef0-8e91-4d98-bf4e-4c8e0574372e" path="/var/lib/kubelet/pods/39b3fef0-8e91-4d98-bf4e-4c8e0574372e/volumes" Oct 30 18:47:27.141246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:27.141235 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aadb74c-066c-483b-9cea-f9114a5a596d" path="/var/lib/kubelet/pods/6aadb74c-066c-483b-9cea-f9114a5a596d/volumes" Oct 30 18:47:27.141949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:27.141934 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" path="/var/lib/kubelet/pods/cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c/volumes" Oct 30 18:47:27.142393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:27.142381 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7797026-78dd-49a6-823d-e904cba88ab1" path="/var/lib/kubelet/pods/e7797026-78dd-49a6-823d-e904cba88ab1/volumes" Oct 30 18:47:27.752506 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:27.752474 5014 scope.go:122] "RemoveContainer" containerID="2f1abcda0b0bb2994c1acc3226069843857520e9bef588f8d435faa3d859b8a6" Oct 30 18:47:34.599705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.599673 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b"] Oct 30 18:47:34.600132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600006 5014 cpu_manager.go:414] "containerMap: removing container" podUID="39107235-3fb6-49af-b295-9be59c67c288" containerName="alertmanager" Oct 30 18:47:34.600132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600023 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39107235-3fb6-49af-b295-9be59c67c288" containerName="alertmanager" Oct 30 18:47:34.600132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600042 5014 cpu_manager.go:414] "containerMap: removing container" podUID="e7797026-78dd-49a6-823d-e904cba88ab1" containerName="kube-rbac-proxy" Oct 30 18:47:34.600132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600048 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7797026-78dd-49a6-823d-e904cba88ab1" containerName="kube-rbac-proxy" Oct 30 18:47:34.600132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600062 5014 cpu_manager.go:414] "containerMap: removing container" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="thanos-sidecar" Oct 30 18:47:34.600132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600092 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="thanos-sidecar" Oct 30 18:47:34.600132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600107 5014 cpu_manager.go:414] "containerMap: removing container" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="config-reloader" Oct 30 18:47:34.600132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600114 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="config-reloader" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600145 5014 cpu_manager.go:414] "containerMap: removing container" podUID="39107235-3fb6-49af-b295-9be59c67c288" containerName="init-config-reloader" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600153 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39107235-3fb6-49af-b295-9be59c67c288" containerName="init-config-reloader" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600163 5014 cpu_manager.go:414] "containerMap: removing container" podUID="39b3fef0-8e91-4d98-bf4e-4c8e0574372e" containerName="otc-container" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600171 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b3fef0-8e91-4d98-bf4e-4c8e0574372e" containerName="otc-container" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600187 5014 cpu_manager.go:414] "containerMap: removing container" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="prometheus" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600196 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="prometheus" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600215 5014 cpu_manager.go:414] "containerMap: removing container" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="init-config-reloader" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600222 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="init-config-reloader" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600236 5014 cpu_manager.go:414] "containerMap: removing container" podUID="6aadb74c-066c-483b-9cea-f9114a5a596d" containerName="thanos-querier" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600243 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aadb74c-066c-483b-9cea-f9114a5a596d" containerName="thanos-querier" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600255 5014 cpu_manager.go:414] "containerMap: removing container" podUID="39107235-3fb6-49af-b295-9be59c67c288" containerName="config-reloader" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600260 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39107235-3fb6-49af-b295-9be59c67c288" containerName="config-reloader" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600382 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="39107235-3fb6-49af-b295-9be59c67c288" containerName="alertmanager" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600397 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7797026-78dd-49a6-823d-e904cba88ab1" containerName="kube-rbac-proxy" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600407 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="thanos-sidecar" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600415 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="config-reloader" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600424 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b3fef0-8e91-4d98-bf4e-4c8e0574372e" containerName="otc-container" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600432 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1fa9ce-20a0-4578-9b1f-92c8f8ecaf7c" containerName="prometheus" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600439 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aadb74c-066c-483b-9cea-f9114a5a596d" containerName="thanos-querier" Oct 30 18:47:34.600477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.600444 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="39107235-3fb6-49af-b295-9be59c67c288" containerName="config-reloader" Oct 30 18:47:34.605775 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.605757 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.622155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.622135 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:47:34.622289 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.622248 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:47:34.622289 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.622250 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:47:34.627509 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.627486 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.627649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.627519 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65vl7\" (UniqueName: \"kubernetes.io/projected/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-api-access-65vl7\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.627649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.627548 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.627649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.627575 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.627850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.627719 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.633463 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.633442 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:47:34.633563 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.633522 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 18:47:34.633563 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.633529 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 18:47:34.633842 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.633816 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-xs2wb\"" Oct 30 18:47:34.635208 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.635184 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b"] Oct 30 18:47:34.691772 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.691747 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z"] Oct 30 18:47:34.695423 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.695406 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:47:34.717343 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.717323 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z"] Oct 30 18:47:34.723048 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.723028 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 18:47:34.723146 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.723097 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 18:47:34.723146 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.723100 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-wqbrg\"" Oct 30 18:47:34.728631 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.728608 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.728740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.728650 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.728740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.728671 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1659057b-2d39-4bc4-ae28-18477a29bf96-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-k294z\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:47:34.728740 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:34.728701 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 18:47:34.728740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.728707 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.728929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.728739 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tptm4\" (UniqueName: \"kubernetes.io/projected/1659057b-2d39-4bc4-ae28-18477a29bf96-kube-api-access-tptm4\") pod \"data-science-thanos-querier-proxy-6998ff8b98-k294z\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:47:34.728929 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:34.728757 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-client-cert podName:73e42dfc-d730-469f-ba46-f3bf3a78803a nodeName:}" failed. No retries permitted until 2026-10-30 18:47:35.228737794 +0000 UTC m=+3068.217078474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" (UID: "73e42dfc-d730-469f-ba46-f3bf3a78803a") : secret "prometheus-operated-tls" not found Oct 30 18:47:34.728929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.728836 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.728929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.728882 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1659057b-2d39-4bc4-ae28-18477a29bf96-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-k294z\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:47:34.728929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.728906 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65vl7\" (UniqueName: \"kubernetes.io/projected/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-api-access-65vl7\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.729684 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.729661 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.731247 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.731229 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.731350 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.731330 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.752415 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.752390 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65vl7\" (UniqueName: \"kubernetes.io/projected/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-api-access-65vl7\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:34.829941 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.829912 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1659057b-2d39-4bc4-ae28-18477a29bf96-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-k294z\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:47:34.830069 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.829979 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1659057b-2d39-4bc4-ae28-18477a29bf96-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-k294z\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:47:34.830069 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.830010 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tptm4\" (UniqueName: \"kubernetes.io/projected/1659057b-2d39-4bc4-ae28-18477a29bf96-kube-api-access-tptm4\") pod \"data-science-thanos-querier-proxy-6998ff8b98-k294z\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:47:34.830676 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.830619 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1659057b-2d39-4bc4-ae28-18477a29bf96-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-k294z\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:47:34.832398 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.832374 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1659057b-2d39-4bc4-ae28-18477a29bf96-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-k294z\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:47:34.861246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:34.861188 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tptm4\" (UniqueName: \"kubernetes.io/projected/1659057b-2d39-4bc4-ae28-18477a29bf96-kube-api-access-tptm4\") pod \"data-science-thanos-querier-proxy-6998ff8b98-k294z\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:47:35.011358 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:35.011336 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:47:35.233122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:35.233043 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:35.235487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:35.235462 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-rvx5b\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:35.309334 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:35.309307 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z"] Oct 30 18:47:35.314056 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:47:35.314030 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1659057b_2d39_4bc4_ae28_18477a29bf96.slice/crio-5303ecf775f012023fcd357ef7c87eceb34bc2ac8649608b38c82d7ea8fbe2cc WatchSource:0}: Error finding container 5303ecf775f012023fcd357ef7c87eceb34bc2ac8649608b38c82d7ea8fbe2cc: Status 404 returned error can't find the container with id 5303ecf775f012023fcd357ef7c87eceb34bc2ac8649608b38c82d7ea8fbe2cc Oct 30 18:47:35.367080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:35.367057 5014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:47:35.517832 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:35.517765 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:35.788698 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:35.788634 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" event={"ID":"1659057b-2d39-4bc4-ae28-18477a29bf96","Type":"ContainerStarted","Data":"efbd58a32baa6642dddc63685def9e91cfc59018f50a7bbd99fa758301aacc2a"} Oct 30 18:47:35.788698 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:35.788698 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" event={"ID":"1659057b-2d39-4bc4-ae28-18477a29bf96","Type":"ContainerStarted","Data":"5303ecf775f012023fcd357ef7c87eceb34bc2ac8649608b38c82d7ea8fbe2cc"} Oct 30 18:47:35.811140 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:35.811113 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b"] Oct 30 18:47:35.815969 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:47:35.815935 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e42dfc_d730_469f_ba46_f3bf3a78803a.slice/crio-a4e4a262efa4c878dde44dc8b5da4e3c340043791a52d426540a5c96257d48ba WatchSource:0}: Error finding container a4e4a262efa4c878dde44dc8b5da4e3c340043791a52d426540a5c96257d48ba: Status 404 returned error can't find the container with id a4e4a262efa4c878dde44dc8b5da4e3c340043791a52d426540a5c96257d48ba Oct 30 18:47:36.043676 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.040017 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:36.049110 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.049061 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.071626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.069038 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:47:36.071626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.069174 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:47:36.071626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.070817 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:36.077103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.076878 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-57l52\"" Oct 30 18:47:36.077103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.076954 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:47:36.090632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.086637 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:47:36.090632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.086722 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:47:36.141478 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.141445 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.141739 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.141496 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.141739 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.141540 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.141739 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.141573 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.141739 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.141625 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbltf\" (UniqueName: \"kubernetes.io/projected/5e02bceb-251d-4507-81ff-0cfd2cea2be4-kube-api-access-kbltf\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.141739 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.141664 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.242656 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.242616 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.242810 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.242679 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.242810 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.242720 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbltf\" (UniqueName: \"kubernetes.io/projected/5e02bceb-251d-4507-81ff-0cfd2cea2be4-kube-api-access-kbltf\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.242810 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.242773 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.242951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.242829 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.242951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.242865 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.244266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.244235 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.247984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.247562 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.247984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.247939 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.248740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.248643 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.249390 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.249363 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.271302 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.271279 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbltf\" (UniqueName: \"kubernetes.io/projected/5e02bceb-251d-4507-81ff-0cfd2cea2be4-kube-api-access-kbltf\") pod \"data-science-collector-collector-0\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.368308 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.368277 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:36.680682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.680423 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:36.684930 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:47:36.684899 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e02bceb_251d_4507_81ff_0cfd2cea2be4.slice/crio-6279b4ba5b711ff4a5636c20ec6810b08e72a389e62ae6cdbd814ec1c55f0d4b WatchSource:0}: Error finding container 6279b4ba5b711ff4a5636c20ec6810b08e72a389e62ae6cdbd814ec1c55f0d4b: Status 404 returned error can't find the container with id 6279b4ba5b711ff4a5636c20ec6810b08e72a389e62ae6cdbd814ec1c55f0d4b Oct 30 18:47:36.688080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.688042 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:36.688193 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.688112 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:36.795947 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.795901 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" event={"ID":"73e42dfc-d730-469f-ba46-f3bf3a78803a","Type":"ContainerStarted","Data":"0afdb6f61782b9b1f8fdab71d97b7bf762290dff6de0d6fd5ac6cd22a662286b"} Oct 30 18:47:36.796415 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.795966 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" event={"ID":"73e42dfc-d730-469f-ba46-f3bf3a78803a","Type":"ContainerStarted","Data":"a4e4a262efa4c878dde44dc8b5da4e3c340043791a52d426540a5c96257d48ba"} Oct 30 18:47:36.796713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.796682 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:36.798646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.798620 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e02bceb-251d-4507-81ff-0cfd2cea2be4","Type":"ContainerStarted","Data":"e884c909d4b1c35b169107d4a66ae55b0d03bf3fb044731db3416ca1bc1904df"} Oct 30 18:47:36.798752 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.798668 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e02bceb-251d-4507-81ff-0cfd2cea2be4","Type":"ContainerStarted","Data":"6279b4ba5b711ff4a5636c20ec6810b08e72a389e62ae6cdbd814ec1c55f0d4b"} Oct 30 18:47:36.837636 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.836487 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" podStartSLOduration=2.836471584 podStartE2EDuration="2.836471584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:36.835826787 +0000 UTC m=+3069.824167275" watchObservedRunningTime="2026-10-30 18:47:36.836471584 +0000 UTC m=+3069.824812072" Oct 30 18:47:36.877874 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:36.877821 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=0.877807337 podStartE2EDuration="877.807337ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:36.876864804 +0000 UTC m=+3069.865205301" watchObservedRunningTime="2026-10-30 18:47:36.877807337 +0000 UTC m=+3069.866148228" Oct 30 18:47:38.812661 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:38.812627 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" event={"ID":"1659057b-2d39-4bc4-ae28-18477a29bf96","Type":"ContainerStarted","Data":"b026b849260300e81d1fbe3fbb9ecfb847167a6a3760a97eef1cde251e8c2f38"} Oct 30 18:47:39.819414 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:39.819379 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" event={"ID":"1659057b-2d39-4bc4-ae28-18477a29bf96","Type":"ContainerStarted","Data":"38487061deda1d1867fe1624354d465e586c76b6a59960770ce18b36304062ba"} Oct 30 18:47:39.861214 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:39.861044 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" podStartSLOduration=2.321234919 podStartE2EDuration="5.861029425s" totalImagesPullingTime="3.539794506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:39.860498599 +0000 UTC m=+3072.848839100" watchObservedRunningTime="2026-10-30 18:47:39.861029425 +0000 UTC m=+3072.849369914" Oct 30 18:47:43.335375 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:47:43.335334 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:47:43.761747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.761672 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:47:43.765894 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.765870 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.785003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.784975 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:47:43.785122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.785109 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:47:43.785248 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.785233 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:47:43.785349 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.785284 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:47:43.796734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.796708 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-b9xnn\"" Oct 30 18:47:43.799015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.798991 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:47:43.807486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.807466 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.807569 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.807505 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.807569 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.807526 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.807718 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.807669 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.807783 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.807711 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnx5\" (UniqueName: \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-kube-api-access-stnx5\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.807855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.807802 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.807855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.807848 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.817253 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.815360 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:47:43.817253 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.816320 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:47:43.821095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.821076 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.846880 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.846846 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:47:43.887299 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.887272 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:47:43.887419 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.887379 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:47:43.887564 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.887547 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:47:43.887564 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.887560 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:47:43.887704 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.887564 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:47:43.887704 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.887554 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-98zpq\"" Oct 30 18:47:43.906885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.906866 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:47:43.906979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.906911 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:47:43.906979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.906950 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:47:43.908452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908432 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnx5\" (UniqueName: \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-kube-api-access-stnx5\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.908537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908471 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.908537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908501 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.908660 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908569 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.908660 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908645 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0225fea7-c461-4952-8e38-33c8c18c7b64-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.908770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908670 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.908770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908691 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.908881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908800 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.908881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908821 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.908881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908840 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.908881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908864 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.909073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.908989 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.909073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.909025 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.909073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.909062 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c7tw\" (UniqueName: \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-kube-api-access-5c7tw\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.909238 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.909091 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.909238 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.909200 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.909353 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.909234 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:43.909353 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.909271 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.909512 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.909432 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.910147 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.910119 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.911371 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.911347 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.911481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.911445 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.911642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.911626 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.911745 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.911728 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.911932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.911916 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:43.933134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:43.933112 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnx5\" (UniqueName: \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-kube-api-access-stnx5\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:44.011088 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011066 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c7tw\" (UniqueName: \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-kube-api-access-5c7tw\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.011199 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011096 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.011199 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011121 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.011199 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011155 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.011199 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011176 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.011199 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011200 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0225fea7-c461-4952-8e38-33c8c18c7b64-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.011481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011225 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.011481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011267 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.011481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011292 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.011481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011333 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.011481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011360 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.011481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.011426 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.012062 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.012002 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.012324 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.012282 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.012634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.012564 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.014046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.014003 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.014190 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.014168 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.014521 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.014498 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0225fea7-c461-4952-8e38-33c8c18c7b64-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.014733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.014711 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.014974 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.014955 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.015639 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.015617 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.015735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.015623 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.015997 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.015981 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.033948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.033927 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c7tw\" (UniqueName: \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-kube-api-access-5c7tw\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:44.083428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.083405 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:44.404462 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.404424 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:47:44.408841 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:47:44.408812 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6748a46_001a_4481_b1aa_bb045eec31ee.slice/crio-c217549fd92ea7a9f57c1acdcb17b6fe9e2062013e7f15336331cfb67a3acb82 WatchSource:0}: Error finding container c217549fd92ea7a9f57c1acdcb17b6fe9e2062013e7f15336331cfb67a3acb82: Status 404 returned error can't find the container with id c217549fd92ea7a9f57c1acdcb17b6fe9e2062013e7f15336331cfb67a3acb82 Oct 30 18:47:44.841605 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.841566 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"b6748a46-001a-4481-b1aa-bb045eec31ee","Type":"ContainerStarted","Data":"81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598"} Oct 30 18:47:44.841747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:44.841660 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"b6748a46-001a-4481-b1aa-bb045eec31ee","Type":"ContainerStarted","Data":"c217549fd92ea7a9f57c1acdcb17b6fe9e2062013e7f15336331cfb67a3acb82"} Oct 30 18:47:48.861397 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:48.861366 5014 generic.go:356] "Generic (PLEG): container finished" podID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerID="81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598" exitCode=0 Oct 30 18:47:48.861909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:48.861439 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"b6748a46-001a-4481-b1aa-bb045eec31ee","Type":"ContainerDied","Data":"81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598"} Oct 30 18:47:49.868344 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:49.868305 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"b6748a46-001a-4481-b1aa-bb045eec31ee","Type":"ContainerStarted","Data":"440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3"} Oct 30 18:47:49.868757 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:49.868374 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"b6748a46-001a-4481-b1aa-bb045eec31ee","Type":"ContainerStarted","Data":"07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e"} Oct 30 18:47:49.868757 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:49.868550 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:49.938658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:49.938614 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.653025328 podStartE2EDuration="6.938575697s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.285550369s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:49.926481273 +0000 UTC m=+3082.914821774" watchObservedRunningTime="2026-10-30 18:47:49.938575697 +0000 UTC m=+3082.926916184" Oct 30 18:47:53.281742 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:53.281706 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:53.309672 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:53.309640 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:53.383087 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:53.383060 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:53.383273 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:53.383205 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:53.397291 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:53.397268 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:53.501165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:53.501132 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/26c702426f1aeab2e9e6116e994cbd362b1230f690bb6d32002e36ee94fa3aec/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:53.528147 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:53.528121 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:53.742632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:53.742580 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:54.072348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:54.072308 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:47:54.077062 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:47:54.077033 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0225fea7_c461_4952_8e38_33c8c18c7b64.slice/crio-af1efe8ce33b5b36e1f320a633e2ce6af9aaf56e6086e85ed0f39fde9113f16e WatchSource:0}: Error finding container af1efe8ce33b5b36e1f320a633e2ce6af9aaf56e6086e85ed0f39fde9113f16e: Status 404 returned error can't find the container with id af1efe8ce33b5b36e1f320a633e2ce6af9aaf56e6086e85ed0f39fde9113f16e Oct 30 18:47:54.892263 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:54.892227 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0225fea7-c461-4952-8e38-33c8c18c7b64","Type":"ContainerStarted","Data":"07fabac5b34d0c2b4e7a3bc17951bd2d5eaa258249f8e4b7d9241532fbc7d61d"} Oct 30 18:47:54.892707 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:54.892298 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0225fea7-c461-4952-8e38-33c8c18c7b64","Type":"ContainerStarted","Data":"af1efe8ce33b5b36e1f320a633e2ce6af9aaf56e6086e85ed0f39fde9113f16e"} Oct 30 18:47:55.874496 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:55.874468 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:47:58.912839 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:58.912803 5014 generic.go:356] "Generic (PLEG): container finished" podID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerID="07fabac5b34d0c2b4e7a3bc17951bd2d5eaa258249f8e4b7d9241532fbc7d61d" exitCode=0 Oct 30 18:47:58.913214 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:58.912847 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0225fea7-c461-4952-8e38-33c8c18c7b64","Type":"ContainerDied","Data":"07fabac5b34d0c2b4e7a3bc17951bd2d5eaa258249f8e4b7d9241532fbc7d61d"} Oct 30 18:47:59.919615 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:59.919558 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0225fea7-c461-4952-8e38-33c8c18c7b64","Type":"ContainerStarted","Data":"b5f2e09b71b5bde851322e26c8e4a2e5338e8d9e2684759c3102be7399105854"} Oct 30 18:47:59.920061 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:59.919655 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0225fea7-c461-4952-8e38-33c8c18c7b64","Type":"ContainerStarted","Data":"9a29219b0fd1bc86c9ba29c605e932485ad7e8728e930f15151d28e64c3ad9e8"} Oct 30 18:47:59.920061 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:59.919696 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0225fea7-c461-4952-8e38-33c8c18c7b64","Type":"ContainerStarted","Data":"72cc0bd16b5f910c6c4c00f89030c2f95ee9dd99166f385a1e792a0717ad3cd4"} Oct 30 18:47:59.996200 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:47:59.996152 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=13.412088937 podStartE2EDuration="16.996140322s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.584051385s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:59.983440674 +0000 UTC m=+3092.971781162" watchObservedRunningTime="2026-10-30 18:47:59.996140322 +0000 UTC m=+3092.984480809" Oct 30 18:48:03.743708 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:03.743677 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:48:08.743322 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:08.743233 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:48:08.748816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:08.748794 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:48:08.966228 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:08.966203 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:48:13.332873 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:48:13.332828 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:48:43.336181 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:48:43.336124 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:48:54.027577 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.027545 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b"] Oct 30 18:48:54.028187 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.027835 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" podUID="73e42dfc-d730-469f-ba46-f3bf3a78803a" containerName="kube-rbac-proxy" containerID="cri-o://0afdb6f61782b9b1f8fdab71d97b7bf762290dff6de0d6fd5ac6cd22a662286b" gracePeriod=30 Oct 30 18:48:54.066017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.065990 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z"] Oct 30 18:48:54.066302 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.066263 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="kube-rbac-proxy" containerID="cri-o://efbd58a32baa6642dddc63685def9e91cfc59018f50a7bbd99fa758301aacc2a" gracePeriod=30 Oct 30 18:48:54.066544 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.066277 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="prom-label-proxy" containerID="cri-o://38487061deda1d1867fe1624354d465e586c76b6a59960770ce18b36304062ba" gracePeriod=30 Oct 30 18:48:54.066544 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.066358 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="method-gate" containerID="cri-o://b026b849260300e81d1fbe3fbb9ecfb847167a6a3760a97eef1cde251e8c2f38" gracePeriod=30 Oct 30 18:48:54.198236 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.198207 5014 generic.go:356] "Generic (PLEG): container finished" podID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerID="38487061deda1d1867fe1624354d465e586c76b6a59960770ce18b36304062ba" exitCode=0 Oct 30 18:48:54.198236 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.198232 5014 generic.go:356] "Generic (PLEG): container finished" podID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerID="b026b849260300e81d1fbe3fbb9ecfb847167a6a3760a97eef1cde251e8c2f38" exitCode=0 Oct 30 18:48:54.198412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.198241 5014 generic.go:356] "Generic (PLEG): container finished" podID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerID="efbd58a32baa6642dddc63685def9e91cfc59018f50a7bbd99fa758301aacc2a" exitCode=0 Oct 30 18:48:54.198412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.198274 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" event={"ID":"1659057b-2d39-4bc4-ae28-18477a29bf96","Type":"ContainerDied","Data":"38487061deda1d1867fe1624354d465e586c76b6a59960770ce18b36304062ba"} Oct 30 18:48:54.198412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.198335 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" event={"ID":"1659057b-2d39-4bc4-ae28-18477a29bf96","Type":"ContainerDied","Data":"b026b849260300e81d1fbe3fbb9ecfb847167a6a3760a97eef1cde251e8c2f38"} Oct 30 18:48:54.198412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.198359 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" event={"ID":"1659057b-2d39-4bc4-ae28-18477a29bf96","Type":"ContainerDied","Data":"efbd58a32baa6642dddc63685def9e91cfc59018f50a7bbd99fa758301aacc2a"} Oct 30 18:48:54.200186 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.200158 5014 generic.go:356] "Generic (PLEG): container finished" podID="73e42dfc-d730-469f-ba46-f3bf3a78803a" containerID="0afdb6f61782b9b1f8fdab71d97b7bf762290dff6de0d6fd5ac6cd22a662286b" exitCode=0 Oct 30 18:48:54.200297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.200191 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" event={"ID":"73e42dfc-d730-469f-ba46-f3bf3a78803a","Type":"ContainerDied","Data":"0afdb6f61782b9b1f8fdab71d97b7bf762290dff6de0d6fd5ac6cd22a662286b"} Oct 30 18:48:54.309002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.308981 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:48:54.335938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.335916 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:48:54.384220 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.384182 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-api-access-65vl7\" (UniqueName: \"kubernetes.io/projected/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-api-access-65vl7\") pod \"73e42dfc-d730-469f-ba46-f3bf3a78803a\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " Oct 30 18:48:54.384417 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.384283 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-client-cert\") pod \"73e42dfc-d730-469f-ba46-f3bf3a78803a\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " Oct 30 18:48:54.384417 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.384315 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-rbac-proxy-config\") pod \"73e42dfc-d730-469f-ba46-f3bf3a78803a\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " Oct 30 18:48:54.384417 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.384353 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-tls\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-tls\") pod \"73e42dfc-d730-469f-ba46-f3bf3a78803a\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " Oct 30 18:48:54.384417 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.384390 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-ca\") pod \"73e42dfc-d730-469f-ba46-f3bf3a78803a\" (UID: \"73e42dfc-d730-469f-ba46-f3bf3a78803a\") " Oct 30 18:48:54.385301 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.385262 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-ca" pod "73e42dfc-d730-469f-ba46-f3bf3a78803a" (UID: "73e42dfc-d730-469f-ba46-f3bf3a78803a"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:54.387243 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.387215 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-client-cert" pod "73e42dfc-d730-469f-ba46-f3bf3a78803a" (UID: "73e42dfc-d730-469f-ba46-f3bf3a78803a"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:54.388817 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.388789 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-api-access-65vl7" pod "73e42dfc-d730-469f-ba46-f3bf3a78803a" (UID: "73e42dfc-d730-469f-ba46-f3bf3a78803a"). InnerVolumeSpecName "kube-api-access-65vl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:54.389348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.389319 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-rbac-proxy-config" pod "73e42dfc-d730-469f-ba46-f3bf3a78803a" (UID: "73e42dfc-d730-469f-ba46-f3bf3a78803a"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:54.389868 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.389848 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-tls" pod "73e42dfc-d730-469f-ba46-f3bf3a78803a" (UID: "73e42dfc-d730-469f-ba46-f3bf3a78803a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.486053 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1659057b-2d39-4bc4-ae28-18477a29bf96-config\" (UniqueName: \"kubernetes.io/configmap/1659057b-2d39-4bc4-ae28-18477a29bf96-config\") pod \"1659057b-2d39-4bc4-ae28-18477a29bf96\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.486207 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1659057b-2d39-4bc4-ae28-18477a29bf96-kube-api-access-tptm4\" (UniqueName: \"kubernetes.io/projected/1659057b-2d39-4bc4-ae28-18477a29bf96-kube-api-access-tptm4\") pod \"1659057b-2d39-4bc4-ae28-18477a29bf96\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.486282 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1659057b-2d39-4bc4-ae28-18477a29bf96-tls\" (UniqueName: \"kubernetes.io/secret/1659057b-2d39-4bc4-ae28-18477a29bf96-tls\") pod \"1659057b-2d39-4bc4-ae28-18477a29bf96\" (UID: \"1659057b-2d39-4bc4-ae28-18477a29bf96\") " Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.486442 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1659057b-2d39-4bc4-ae28-18477a29bf96-config" pod "1659057b-2d39-4bc4-ae28-18477a29bf96" (UID: "1659057b-2d39-4bc4-ae28-18477a29bf96"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.486623 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1659057b-2d39-4bc4-ae28-18477a29bf96-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.486642 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.486657 5014 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-rbac-proxy-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.486671 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73e42dfc-d730-469f-ba46-f3bf3a78803a-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.486686 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/73e42dfc-d730-469f-ba46-f3bf3a78803a-prometheus-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.486700 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65vl7\" (UniqueName: \"kubernetes.io/projected/73e42dfc-d730-469f-ba46-f3bf3a78803a-kube-api-access-65vl7\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.489058 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1659057b-2d39-4bc4-ae28-18477a29bf96-kube-api-access-tptm4" pod "1659057b-2d39-4bc4-ae28-18477a29bf96" (UID: "1659057b-2d39-4bc4-ae28-18477a29bf96"). InnerVolumeSpecName "kube-api-access-tptm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:54.490625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.489681 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1659057b-2d39-4bc4-ae28-18477a29bf96-tls" pod "1659057b-2d39-4bc4-ae28-18477a29bf96" (UID: "1659057b-2d39-4bc4-ae28-18477a29bf96"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:54.520369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.520340 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:48:54.520742 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.520707 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="alertmanager" containerID="cri-o://07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e" gracePeriod=120 Oct 30 18:48:54.521154 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.520743 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="config-reloader" containerID="cri-o://440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3" gracePeriod=120 Oct 30 18:48:54.523355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.523310 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:48:54.523762 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.523631 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="prometheus" containerID="cri-o://72cc0bd16b5f910c6c4c00f89030c2f95ee9dd99166f385a1e792a0717ad3cd4" gracePeriod=600 Oct 30 18:48:54.523762 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.523665 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="config-reloader" containerID="cri-o://9a29219b0fd1bc86c9ba29c605e932485ad7e8728e930f15151d28e64c3ad9e8" gracePeriod=600 Oct 30 18:48:54.523762 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.523731 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="thanos-sidecar" containerID="cri-o://b5f2e09b71b5bde851322e26c8e4a2e5338e8d9e2684759c3102be7399105854" gracePeriod=600 Oct 30 18:48:54.587117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.587091 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tptm4\" (UniqueName: \"kubernetes.io/projected/1659057b-2d39-4bc4-ae28-18477a29bf96-kube-api-access-tptm4\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.587117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:54.587117 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1659057b-2d39-4bc4-ae28-18477a29bf96-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.085738 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.085702 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:48:55.086229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.085981 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5e02bceb-251d-4507-81ff-0cfd2cea2be4" containerName="otc-container" containerID="cri-o://e884c909d4b1c35b169107d4a66ae55b0d03bf3fb044731db3416ca1bc1904df" gracePeriod=30 Oct 30 18:48:55.209114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.209088 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" Oct 30 18:48:55.209114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.209092 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z" event={"ID":"1659057b-2d39-4bc4-ae28-18477a29bf96","Type":"ContainerDied","Data":"5303ecf775f012023fcd357ef7c87eceb34bc2ac8649608b38c82d7ea8fbe2cc"} Oct 30 18:48:55.209304 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.209207 5014 scope.go:122] "RemoveContainer" containerID="38487061deda1d1867fe1624354d465e586c76b6a59960770ce18b36304062ba" Oct 30 18:48:55.213786 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.213629 5014 generic.go:356] "Generic (PLEG): container finished" podID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerID="440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3" exitCode=0 Oct 30 18:48:55.213786 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.213723 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"b6748a46-001a-4481-b1aa-bb045eec31ee","Type":"ContainerDied","Data":"440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3"} Oct 30 18:48:55.222297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.222072 5014 scope.go:122] "RemoveContainer" containerID="b026b849260300e81d1fbe3fbb9ecfb847167a6a3760a97eef1cde251e8c2f38" Oct 30 18:48:55.227902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.227225 5014 generic.go:356] "Generic (PLEG): container finished" podID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerID="b5f2e09b71b5bde851322e26c8e4a2e5338e8d9e2684759c3102be7399105854" exitCode=0 Oct 30 18:48:55.227902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.227245 5014 generic.go:356] "Generic (PLEG): container finished" podID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerID="9a29219b0fd1bc86c9ba29c605e932485ad7e8728e930f15151d28e64c3ad9e8" exitCode=0 Oct 30 18:48:55.227902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.227257 5014 generic.go:356] "Generic (PLEG): container finished" podID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerID="72cc0bd16b5f910c6c4c00f89030c2f95ee9dd99166f385a1e792a0717ad3cd4" exitCode=0 Oct 30 18:48:55.227902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.227370 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0225fea7-c461-4952-8e38-33c8c18c7b64","Type":"ContainerDied","Data":"b5f2e09b71b5bde851322e26c8e4a2e5338e8d9e2684759c3102be7399105854"} Oct 30 18:48:55.227902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.227433 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0225fea7-c461-4952-8e38-33c8c18c7b64","Type":"ContainerDied","Data":"9a29219b0fd1bc86c9ba29c605e932485ad7e8728e930f15151d28e64c3ad9e8"} Oct 30 18:48:55.227902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.227471 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0225fea7-c461-4952-8e38-33c8c18c7b64","Type":"ContainerDied","Data":"72cc0bd16b5f910c6c4c00f89030c2f95ee9dd99166f385a1e792a0717ad3cd4"} Oct 30 18:48:55.238919 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.234449 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" event={"ID":"73e42dfc-d730-469f-ba46-f3bf3a78803a","Type":"ContainerDied","Data":"a4e4a262efa4c878dde44dc8b5da4e3c340043791a52d426540a5c96257d48ba"} Oct 30 18:48:55.238919 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.234531 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b" Oct 30 18:48:55.238919 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.235681 5014 scope.go:122] "RemoveContainer" containerID="efbd58a32baa6642dddc63685def9e91cfc59018f50a7bbd99fa758301aacc2a" Oct 30 18:48:55.238919 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.238214 5014 generic.go:356] "Generic (PLEG): container finished" podID="5e02bceb-251d-4507-81ff-0cfd2cea2be4" containerID="e884c909d4b1c35b169107d4a66ae55b0d03bf3fb044731db3416ca1bc1904df" exitCode=0 Oct 30 18:48:55.238919 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.238276 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e02bceb-251d-4507-81ff-0cfd2cea2be4","Type":"ContainerDied","Data":"e884c909d4b1c35b169107d4a66ae55b0d03bf3fb044731db3416ca1bc1904df"} Oct 30 18:48:55.246641 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.246620 5014 scope.go:122] "RemoveContainer" containerID="0afdb6f61782b9b1f8fdab71d97b7bf762290dff6de0d6fd5ac6cd22a662286b" Oct 30 18:48:55.266583 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.266553 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z"] Oct 30 18:48:55.269888 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.269854 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-k294z"] Oct 30 18:48:55.304495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.303463 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b"] Oct 30 18:48:55.307352 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.307324 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-rvx5b"] Oct 30 18:48:55.389155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.389132 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:48:55.391184 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.391169 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:48:55.493461 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493426 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.493646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493487 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.493646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493516 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-tls-certs\") pod \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " Oct 30 18:48:55.493646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493540 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otc-internal\") pod \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " Oct 30 18:48:55.493646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493566 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-config\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-config\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.493646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493619 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-secret-prometheus-web-tls-ca\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493656 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-thanos-prometheus-http-client-file\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493783 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493817 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493849 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-kube-api-access-5c7tw\" (UniqueName: \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-kube-api-access-5c7tw\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493878 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-monitoring-tls-certs\") pod \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493883 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-1" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493908 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0225fea7-c461-4952-8e38-33c8c18c7b64-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0225fea7-c461-4952-8e38-33c8c18c7b64-config-out\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493949 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.493976 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.494044 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e02bceb-251d-4507-81ff-0cfd2cea2be4-kube-api-access-kbltf\" (UniqueName: \"kubernetes.io/projected/5e02bceb-251d-4507-81ff-0cfd2cea2be4-kube-api-access-kbltf\") pod \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.494070 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-tls-assets\" (UniqueName: \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-tls-assets\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.494116 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"0225fea7-c461-4952-8e38-33c8c18c7b64\" (UID: \"0225fea7-c461-4952-8e38-33c8c18c7b64\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.494145 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-data-science-collector-ta-client-cert\") pod \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " Oct 30 18:48:55.494264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.494203 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otlp-tls-certs\") pod \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\" (UID: \"5e02bceb-251d-4507-81ff-0cfd2cea2be4\") " Oct 30 18:48:55.495767 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.494489 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.495767 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.495212 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otc-internal" pod "5e02bceb-251d-4507-81ff-0cfd2cea2be4" (UID: "5e02bceb-251d-4507-81ff-0cfd2cea2be4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:55.495910 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.495817 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-0" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:55.497192 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.496929 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-2" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:55.499233 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.499204 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-tls-certs" pod "5e02bceb-251d-4507-81ff-0cfd2cea2be4" (UID: "5e02bceb-251d-4507-81ff-0cfd2cea2be4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.499345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.499232 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-monitoring-tls-certs" pod "5e02bceb-251d-4507-81ff-0cfd2cea2be4" (UID: "5e02bceb-251d-4507-81ff-0cfd2cea2be4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.499681 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.499641 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otlp-tls-certs" pod "5e02bceb-251d-4507-81ff-0cfd2cea2be4" (UID: "5e02bceb-251d-4507-81ff-0cfd2cea2be4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.500778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.500754 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e02bceb-251d-4507-81ff-0cfd2cea2be4-kube-api-access-kbltf" pod "5e02bceb-251d-4507-81ff-0cfd2cea2be4" (UID: "5e02bceb-251d-4507-81ff-0cfd2cea2be4"). InnerVolumeSpecName "kube-api-access-kbltf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:55.500778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.500900 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-thanos-prometheus-http-client-file" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.501143 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.501062 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-kube-api-access-5c7tw" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "kube-api-access-5c7tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:55.501553 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.501519 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.501772 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.501671 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-tls-assets" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:55.501847 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.501770 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0225fea7-c461-4952-8e38-33c8c18c7b64-config-out" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:55.502005 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.501981 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-secret-prometheus-web-tls-ca" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.502252 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.502226 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.503375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.503350 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-data-science-collector-ta-client-cert" pod "5e02bceb-251d-4507-81ff-0cfd2cea2be4" (UID: "5e02bceb-251d-4507-81ff-0cfd2cea2be4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.503499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.503483 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-config" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.505444 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.505425 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "pvc-c02e8472-bb1a-4974-aa89-86eef715889b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:48:55.515071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.515050 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config" pod "0225fea7-c461-4952-8e38-33c8c18c7b64" (UID: "0225fea7-c461-4952-8e38-33c8c18c7b64"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.595086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595020 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595057 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595076 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595089 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595104 5014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595117 5014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595157 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595173 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595186 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c7tw\" (UniqueName: \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-kube-api-access-5c7tw\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595194 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595202 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0225fea7-c461-4952-8e38-33c8c18c7b64-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595211 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595219 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0225fea7-c461-4952-8e38-33c8c18c7b64-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595227 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbltf\" (UniqueName: \"kubernetes.io/projected/5e02bceb-251d-4507-81ff-0cfd2cea2be4-kube-api-access-kbltf\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595234 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0225fea7-c461-4952-8e38-33c8c18c7b64-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595243 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0225fea7-c461-4952-8e38-33c8c18c7b64-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595252 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.595710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.595266 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e02bceb-251d-4507-81ff-0cfd2cea2be4-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.606497 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.606483 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c02e8472-bb1a-4974-aa89-86eef715889b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:48:55.696412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.696387 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" Oct 30 18:48:55.873926 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:55.873857 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.120:9093/-/ready\": dial tcp 10.132.0.120:9093: connect: connection refused" Oct 30 18:48:56.245443 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.245369 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:48:56.245443 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.245379 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0225fea7-c461-4952-8e38-33c8c18c7b64","Type":"ContainerDied","Data":"af1efe8ce33b5b36e1f320a633e2ce6af9aaf56e6086e85ed0f39fde9113f16e"} Oct 30 18:48:56.245976 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.245451 5014 scope.go:122] "RemoveContainer" containerID="b5f2e09b71b5bde851322e26c8e4a2e5338e8d9e2684759c3102be7399105854" Oct 30 18:48:56.247902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.247885 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:48:56.247992 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.247914 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e02bceb-251d-4507-81ff-0cfd2cea2be4","Type":"ContainerDied","Data":"6279b4ba5b711ff4a5636c20ec6810b08e72a389e62ae6cdbd814ec1c55f0d4b"} Oct 30 18:48:56.255010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.254994 5014 scope.go:122] "RemoveContainer" containerID="9a29219b0fd1bc86c9ba29c605e932485ad7e8728e930f15151d28e64c3ad9e8" Oct 30 18:48:56.263069 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.263053 5014 scope.go:122] "RemoveContainer" containerID="72cc0bd16b5f910c6c4c00f89030c2f95ee9dd99166f385a1e792a0717ad3cd4" Oct 30 18:48:56.271175 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.271152 5014 scope.go:122] "RemoveContainer" containerID="07fabac5b34d0c2b4e7a3bc17951bd2d5eaa258249f8e4b7d9241532fbc7d61d" Oct 30 18:48:56.278991 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.278975 5014 scope.go:122] "RemoveContainer" containerID="e884c909d4b1c35b169107d4a66ae55b0d03bf3fb044731db3416ca1bc1904df" Oct 30 18:48:56.297820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.297801 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:48:56.309160 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.309141 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:48:56.336086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.336066 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:48:56.340484 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:56.340466 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:48:57.142750 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:57.142717 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" path="/var/lib/kubelet/pods/0225fea7-c461-4952-8e38-33c8c18c7b64/volumes" Oct 30 18:48:57.143227 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:57.143213 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" path="/var/lib/kubelet/pods/1659057b-2d39-4bc4-ae28-18477a29bf96/volumes" Oct 30 18:48:57.143892 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:57.143876 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e02bceb-251d-4507-81ff-0cfd2cea2be4" path="/var/lib/kubelet/pods/5e02bceb-251d-4507-81ff-0cfd2cea2be4/volumes" Oct 30 18:48:57.144227 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:48:57.144216 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e42dfc-d730-469f-ba46-f3bf3a78803a" path="/var/lib/kubelet/pods/73e42dfc-d730-469f-ba46-f3bf3a78803a/volumes" Oct 30 18:49:00.873927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:00.873886 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.120:9093/-/ready\": dial tcp 10.132.0.120:9093: connect: connection refused" Oct 30 18:49:04.776153 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.776127 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:49:04.864901 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.864875 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-alertmanager-data-science-monitoringstack-db\") pod \"b6748a46-001a-4481-b1aa-bb045eec31ee\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " Oct 30 18:49:04.865094 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.864943 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-cluster-tls-config\") pod \"b6748a46-001a-4481-b1aa-bb045eec31ee\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " Oct 30 18:49:04.865094 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.864997 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-config-out\") pod \"b6748a46-001a-4481-b1aa-bb045eec31ee\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " Oct 30 18:49:04.865094 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.865034 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-tls-assets\" (UniqueName: \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-tls-assets\") pod \"b6748a46-001a-4481-b1aa-bb045eec31ee\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " Oct 30 18:49:04.865094 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.865071 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-config-volume\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-config-volume\") pod \"b6748a46-001a-4481-b1aa-bb045eec31ee\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " Oct 30 18:49:04.865305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.865095 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-web-config\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-web-config\") pod \"b6748a46-001a-4481-b1aa-bb045eec31ee\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " Oct 30 18:49:04.865305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.865123 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-kube-api-access-stnx5\" (UniqueName: \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-kube-api-access-stnx5\") pod \"b6748a46-001a-4481-b1aa-bb045eec31ee\" (UID: \"b6748a46-001a-4481-b1aa-bb045eec31ee\") " Oct 30 18:49:04.865305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.865252 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-alertmanager-data-science-monitoringstack-db" pod "b6748a46-001a-4481-b1aa-bb045eec31ee" (UID: "b6748a46-001a-4481-b1aa-bb045eec31ee"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:49:04.865436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.865410 5014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:49:04.867394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.867355 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-config-volume" pod "b6748a46-001a-4481-b1aa-bb045eec31ee" (UID: "b6748a46-001a-4481-b1aa-bb045eec31ee"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:04.867536 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.867484 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-kube-api-access-stnx5" pod "b6748a46-001a-4481-b1aa-bb045eec31ee" (UID: "b6748a46-001a-4481-b1aa-bb045eec31ee"). InnerVolumeSpecName "kube-api-access-stnx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:04.867783 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.867756 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-config-out" pod "b6748a46-001a-4481-b1aa-bb045eec31ee" (UID: "b6748a46-001a-4481-b1aa-bb045eec31ee"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:49:04.867920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.867901 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-tls-assets" pod "b6748a46-001a-4481-b1aa-bb045eec31ee" (UID: "b6748a46-001a-4481-b1aa-bb045eec31ee"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:04.871949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.871922 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-cluster-tls-config" pod "b6748a46-001a-4481-b1aa-bb045eec31ee" (UID: "b6748a46-001a-4481-b1aa-bb045eec31ee"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:04.879659 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.879639 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-web-config" pod "b6748a46-001a-4481-b1aa-bb045eec31ee" (UID: "b6748a46-001a-4481-b1aa-bb045eec31ee"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:04.966258 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.966230 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6748a46-001a-4481-b1aa-bb045eec31ee-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:49:04.966358 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.966261 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:49:04.966358 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.966277 5014 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-config-volume\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:49:04.966358 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.966317 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:49:04.966358 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.966334 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stnx5\" (UniqueName: \"kubernetes.io/projected/b6748a46-001a-4481-b1aa-bb045eec31ee-kube-api-access-stnx5\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:49:04.966358 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:04.966349 5014 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6748a46-001a-4481-b1aa-bb045eec31ee-cluster-tls-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:49:05.292968 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.292936 5014 generic.go:356] "Generic (PLEG): container finished" podID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerID="07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e" exitCode=0 Oct 30 18:49:05.293152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.293013 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"b6748a46-001a-4481-b1aa-bb045eec31ee","Type":"ContainerDied","Data":"07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e"} Oct 30 18:49:05.293152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.293061 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"b6748a46-001a-4481-b1aa-bb045eec31ee","Type":"ContainerDied","Data":"c217549fd92ea7a9f57c1acdcb17b6fe9e2062013e7f15336331cfb67a3acb82"} Oct 30 18:49:05.293152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.293085 5014 scope.go:122] "RemoveContainer" containerID="440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3" Oct 30 18:49:05.293152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.293014 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:49:05.301421 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.301404 5014 scope.go:122] "RemoveContainer" containerID="07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e" Oct 30 18:49:05.309138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.309120 5014 scope.go:122] "RemoveContainer" containerID="81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598" Oct 30 18:49:05.316317 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.316298 5014 scope.go:122] "RemoveContainer" containerID="440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3" Oct 30 18:49:05.316563 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:49:05.316542 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3\": container with ID starting with 440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3 not found: ID does not exist" containerID="440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3" Oct 30 18:49:05.316652 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.316571 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3"} err="failed to get container status \"440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3\": rpc error: code = NotFound desc = could not find container \"440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3\": container with ID starting with 440b807430dcf77028b369654642d6abcbc06475b66ab6fe4d1318fb65685ed3 not found: ID does not exist" Oct 30 18:49:05.316652 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.316586 5014 scope.go:122] "RemoveContainer" containerID="07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e" Oct 30 18:49:05.316867 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:49:05.316847 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e\": container with ID starting with 07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e not found: ID does not exist" containerID="07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e" Oct 30 18:49:05.316926 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.316878 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e"} err="failed to get container status \"07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e\": rpc error: code = NotFound desc = could not find container \"07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e\": container with ID starting with 07fcc432ec3febbb29373202ef07ba69c26f2579f56aafb1bbd17b7402ee0a4e not found: ID does not exist" Oct 30 18:49:05.316926 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.316902 5014 scope.go:122] "RemoveContainer" containerID="81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598" Oct 30 18:49:05.317178 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:49:05.317161 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598\": container with ID starting with 81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598 not found: ID does not exist" containerID="81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598" Oct 30 18:49:05.317246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.317185 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598"} err="failed to get container status \"81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598\": rpc error: code = NotFound desc = could not find container \"81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598\": container with ID starting with 81e72a245f2fadd4fda600530e7b93b9471c198d04fc6e04598f88361a6e7598 not found: ID does not exist" Oct 30 18:49:05.346943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.346925 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:49:05.354978 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:05.354959 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:49:07.141009 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:07.140972 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" path="/var/lib/kubelet/pods/b6748a46-001a-4481-b1aa-bb045eec31ee/volumes" Oct 30 18:49:09.743431 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743399 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743730 5014 cpu_manager.go:414] "containerMap: removing container" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="method-gate" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743742 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="method-gate" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743758 5014 cpu_manager.go:414] "containerMap: removing container" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="config-reloader" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743763 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="config-reloader" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743770 5014 cpu_manager.go:414] "containerMap: removing container" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="thanos-sidecar" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743775 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="thanos-sidecar" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743785 5014 cpu_manager.go:414] "containerMap: removing container" podUID="73e42dfc-d730-469f-ba46-f3bf3a78803a" containerName="kube-rbac-proxy" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743789 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e42dfc-d730-469f-ba46-f3bf3a78803a" containerName="kube-rbac-proxy" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743797 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="alertmanager" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743802 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="alertmanager" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743811 5014 cpu_manager.go:414] "containerMap: removing container" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="kube-rbac-proxy" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743815 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="kube-rbac-proxy" Oct 30 18:49:09.743816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743823 5014 cpu_manager.go:414] "containerMap: removing container" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="init-config-reloader" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743828 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="init-config-reloader" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743841 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="config-reloader" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743846 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="config-reloader" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743856 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="init-config-reloader" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743860 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="init-config-reloader" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743866 5014 cpu_manager.go:414] "containerMap: removing container" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="prometheus" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743872 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="prometheus" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743878 5014 cpu_manager.go:414] "containerMap: removing container" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="prom-label-proxy" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743883 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="prom-label-proxy" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743892 5014 cpu_manager.go:414] "containerMap: removing container" podUID="5e02bceb-251d-4507-81ff-0cfd2cea2be4" containerName="otc-container" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743896 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e02bceb-251d-4507-81ff-0cfd2cea2be4" containerName="otc-container" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743958 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e42dfc-d730-469f-ba46-f3bf3a78803a" containerName="kube-rbac-proxy" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743964 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="alertmanager" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743972 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="kube-rbac-proxy" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743978 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6748a46-001a-4481-b1aa-bb045eec31ee" containerName="config-reloader" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743984 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="prometheus" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743989 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="prom-label-proxy" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.743995 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e02bceb-251d-4507-81ff-0cfd2cea2be4" containerName="otc-container" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.744001 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1659057b-2d39-4bc4-ae28-18477a29bf96" containerName="method-gate" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.744008 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="config-reloader" Oct 30 18:49:09.744271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.744013 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0225fea7-c461-4952-8e38-33c8c18c7b64" containerName="thanos-sidecar" Oct 30 18:49:09.748606 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.748576 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:49:09.763335 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.763305 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 18:49:09.773606 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.773574 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-5mtd6\"" Oct 30 18:49:09.773705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.773615 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:49:09.773705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.773686 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:49:09.907090 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:09.907062 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqcw7\" (UniqueName: \"kubernetes.io/projected/5f359add-07be-44c3-8214-1942f5846ae7-kube-api-access-nqcw7\") pod \"seaweedfs-bucket-creator\" (UID: \"5f359add-07be-44c3-8214-1942f5846ae7\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:49:10.008168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:10.008101 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqcw7\" (UniqueName: \"kubernetes.io/projected/5f359add-07be-44c3-8214-1942f5846ae7-kube-api-access-nqcw7\") pod \"seaweedfs-bucket-creator\" (UID: \"5f359add-07be-44c3-8214-1942f5846ae7\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:49:10.031371 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:10.031349 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqcw7\" (UniqueName: \"kubernetes.io/projected/5f359add-07be-44c3-8214-1942f5846ae7-kube-api-access-nqcw7\") pod \"seaweedfs-bucket-creator\" (UID: \"5f359add-07be-44c3-8214-1942f5846ae7\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:49:10.071995 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:10.071972 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:49:10.363458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:10.363426 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 18:49:10.368152 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:49:10.368125 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f359add_07be_44c3_8214_1942f5846ae7.slice/crio-b3f9df61e3d7afda7da9254c1facffd321c9883d99cdc3f0288cb39f7b932259 WatchSource:0}: Error finding container b3f9df61e3d7afda7da9254c1facffd321c9883d99cdc3f0288cb39f7b932259: Status 404 returned error can't find the container with id b3f9df61e3d7afda7da9254c1facffd321c9883d99cdc3f0288cb39f7b932259 Oct 30 18:49:11.321475 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:11.321445 5014 generic.go:356] "Generic (PLEG): container finished" podID="5f359add-07be-44c3-8214-1942f5846ae7" containerID="68d8b4fbf071815d809150349b2e4d86428bac1251b9927928b3bbb3b0a09ed3" exitCode=0 Oct 30 18:49:11.321916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:11.321522 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"5f359add-07be-44c3-8214-1942f5846ae7","Type":"ContainerDied","Data":"68d8b4fbf071815d809150349b2e4d86428bac1251b9927928b3bbb3b0a09ed3"} Oct 30 18:49:11.321916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:11.321558 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"5f359add-07be-44c3-8214-1942f5846ae7","Type":"ContainerStarted","Data":"b3f9df61e3d7afda7da9254c1facffd321c9883d99cdc3f0288cb39f7b932259"} Oct 30 18:49:12.462570 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:12.462541 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:49:12.627054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:12.626977 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f359add-07be-44c3-8214-1942f5846ae7-kube-api-access-nqcw7\" (UniqueName: \"kubernetes.io/projected/5f359add-07be-44c3-8214-1942f5846ae7-kube-api-access-nqcw7\") pod \"5f359add-07be-44c3-8214-1942f5846ae7\" (UID: \"5f359add-07be-44c3-8214-1942f5846ae7\") " Oct 30 18:49:12.628987 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:12.628962 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f359add-07be-44c3-8214-1942f5846ae7-kube-api-access-nqcw7" pod "5f359add-07be-44c3-8214-1942f5846ae7" (UID: "5f359add-07be-44c3-8214-1942f5846ae7"). InnerVolumeSpecName "kube-api-access-nqcw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:12.728464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:12.728433 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqcw7\" (UniqueName: \"kubernetes.io/projected/5f359add-07be-44c3-8214-1942f5846ae7-kube-api-access-nqcw7\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:49:13.331752 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:13.331713 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"5f359add-07be-44c3-8214-1942f5846ae7","Type":"ContainerDied","Data":"b3f9df61e3d7afda7da9254c1facffd321c9883d99cdc3f0288cb39f7b932259"} Oct 30 18:49:13.331752 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:13.331730 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 18:49:13.331752 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:13.331752 5014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3f9df61e3d7afda7da9254c1facffd321c9883d99cdc3f0288cb39f7b932259" Oct 30 18:49:13.332451 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:49:13.332422 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:49:16.448269 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.448234 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr"] Oct 30 18:49:16.448949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.448624 5014 cpu_manager.go:414] "containerMap: removing container" podUID="5f359add-07be-44c3-8214-1942f5846ae7" containerName="create-bucket" Oct 30 18:49:16.448949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.448653 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f359add-07be-44c3-8214-1942f5846ae7" containerName="create-bucket" Oct 30 18:49:16.448949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.448748 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f359add-07be-44c3-8214-1942f5846ae7" containerName="create-bucket" Oct 30 18:49:16.452074 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.452049 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.477783 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.477756 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-distributor-http\"" Oct 30 18:49:16.477935 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.477846 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-distributor-grpc\"" Oct 30 18:49:16.477999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.477927 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:49:16.477999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.477943 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-dockercfg-6pb2l\"" Oct 30 18:49:16.478164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.478144 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:49:16.478277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.478171 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ca-bundle\"" Oct 30 18:49:16.478588 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.478569 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-config\"" Oct 30 18:49:16.479914 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.479895 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr"] Oct 30 18:49:16.557371 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.557346 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd44r\" (UniqueName: \"kubernetes.io/projected/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-kube-api-access-nd44r\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.557461 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.557379 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.557461 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.557401 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.557544 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.557512 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.557584 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.557549 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.658868 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.658845 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.658984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.658890 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd44r\" (UniqueName: \"kubernetes.io/projected/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-kube-api-access-nd44r\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.658984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.658911 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.659086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.659041 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.659143 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.659093 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.659823 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.659742 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.660041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.659966 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.661485 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.661462 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.661776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.661742 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.681931 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.681908 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd44r\" (UniqueName: \"kubernetes.io/projected/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-kube-api-access-nd44r\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:16.772518 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:16.772453 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:17.073915 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.073883 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr"] Oct 30 18:49:17.078898 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:49:17.078873 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1a87f79_8c4e_416a_9f43_f00e8a29c34f.slice/crio-650d5e9cfd752ae82f1fb1240f2545b5fd128622c3d42a6c12f711987a3a03db WatchSource:0}: Error finding container 650d5e9cfd752ae82f1fb1240f2545b5fd128622c3d42a6c12f711987a3a03db: Status 404 returned error can't find the container with id 650d5e9cfd752ae82f1fb1240f2545b5fd128622c3d42a6c12f711987a3a03db Oct 30 18:49:17.165529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.165507 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4"] Oct 30 18:49:17.171578 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.171550 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.187248 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.187227 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 18:49:17.190224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.190205 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-ca-bundle\"" Oct 30 18:49:17.190479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.190447 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 18:49:17.190479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.190493 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-http\"" Oct 30 18:49:17.190831 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.190811 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-dockercfg-9wphs\"" Oct 30 18:49:17.191491 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.191471 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-client-http\"" Oct 30 18:49:17.192264 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.192180 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4"] Oct 30 18:49:17.353793 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.353714 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" event={"ID":"f1a87f79-8c4e-416a-9f43-f00e8a29c34f","Type":"ContainerStarted","Data":"650d5e9cfd752ae82f1fb1240f2545b5fd128622c3d42a6c12f711987a3a03db"} Oct 30 18:49:17.366163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.366136 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.366278 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.366172 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.366278 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.366206 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tenants\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.366363 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.366315 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-rbac\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.366363 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.366351 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.366426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.366369 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.366426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.366411 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.366494 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.366455 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j48r2\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-kube-api-access-j48r2\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.467291 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.467261 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.467755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.467304 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.467755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.467331 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tenants\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.467755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.467361 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-rbac\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.467755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.467383 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.467755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.467401 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.467755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.467434 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.467755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.467466 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j48r2\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-kube-api-access-j48r2\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.468335 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.468105 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.468335 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.468276 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-rbac\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.468666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.468575 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.469951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.469930 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.470049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.470015 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.470296 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.470278 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tenants\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.470349 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.470334 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.490329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.490308 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j48r2\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-kube-api-access-j48r2\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:17.786734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:17.786682 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:18.111939 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:18.111915 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4"] Oct 30 18:49:18.116402 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:49:18.116375 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod993c6387_cd33_4c37_b5e9_1e3ac65852c7.slice/crio-5ddc8e9092ef45f916bd1f5907cf8d245ece39606f2eb1f2e19b62d561531c5d WatchSource:0}: Error finding container 5ddc8e9092ef45f916bd1f5907cf8d245ece39606f2eb1f2e19b62d561531c5d: Status 404 returned error can't find the container with id 5ddc8e9092ef45f916bd1f5907cf8d245ece39606f2eb1f2e19b62d561531c5d Oct 30 18:49:18.363801 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:18.363722 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" event={"ID":"993c6387-cd33-4c37-b5e9-1e3ac65852c7","Type":"ContainerStarted","Data":"5ddc8e9092ef45f916bd1f5907cf8d245ece39606f2eb1f2e19b62d561531c5d"} Oct 30 18:49:19.854285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.854202 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:49:19.861281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.861258 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:19.890178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.890148 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-http\"" Oct 30 18:49:19.890178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.890148 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-usage-logs-loki-secret\"" Oct 30 18:49:19.890469 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.890433 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-grpc\"" Oct 30 18:49:19.892700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.892673 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:49:19.995699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.995664 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:19.995876 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.995716 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:19.995954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.995868 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:19.995954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.995929 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:19.996077 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.995982 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:19.996077 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:19.996021 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frm6g\" (UniqueName: \"kubernetes.io/projected/a2ba3211-cf44-412e-afad-12b8f5c792bd-kube-api-access-frm6g\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.097580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.096692 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.097580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.096757 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frm6g\" (UniqueName: \"kubernetes.io/projected/a2ba3211-cf44-412e-afad-12b8f5c792bd-kube-api-access-frm6g\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.097580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.096807 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.097580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.096848 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.097580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.096926 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.097580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.096983 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.098527 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.098491 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.099378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.099350 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.100864 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.100835 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.101076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.101024 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.101194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.101132 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:20.120570 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:20.120509 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frm6g\" (UniqueName: \"kubernetes.io/projected/a2ba3211-cf44-412e-afad-12b8f5c792bd-kube-api-access-frm6g\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:21.388739 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:21.388706 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" event={"ID":"993c6387-cd33-4c37-b5e9-1e3ac65852c7","Type":"ContainerStarted","Data":"75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5"} Oct 30 18:49:21.390188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:21.390163 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" event={"ID":"f1a87f79-8c4e-416a-9f43-f00e8a29c34f","Type":"ContainerStarted","Data":"599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51"} Oct 30 18:49:21.390337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:21.390285 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:21.432796 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:21.432757 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" podStartSLOduration=1.595050068 podStartE2EDuration="5.432745118s" totalImagesPullingTime="3.83769505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:21.432375643 +0000 UTC m=+3174.420716133" watchObservedRunningTime="2026-10-30 18:49:21.432745118 +0000 UTC m=+3174.421085598" Oct 30 18:49:23.401212 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:23.401184 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" event={"ID":"993c6387-cd33-4c37-b5e9-1e3ac65852c7","Type":"ContainerStarted","Data":"6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a"} Oct 30 18:49:23.401634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:23.401306 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:23.401634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:23.401342 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:23.407153 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:23.407135 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:23.410206 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:23.410190 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:49:23.469696 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:23.469643 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" podStartSLOduration=1.755063087 podStartE2EDuration="6.46962808s" totalImagesPullingTime="4.714564993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:23.457069517 +0000 UTC m=+3176.445410003" watchObservedRunningTime="2026-10-30 18:49:23.46962808 +0000 UTC m=+3176.457968575" Oct 30 18:49:25.640685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:25.640655 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:25.669543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:25.669506 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:25.741350 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:25.741328 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:25.741471 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:25.741421 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:25.755318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:25.755292 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") DevicePath \"csi-25f379426d8c7fcf77dc6a7d8d83d8aad1de842552d7dbf772ddd741691b9210\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:25.983214 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:25.983187 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/90f68065c219b046956a3eedc2bf9d54a167b2b54b7bb678301a6348f6107900/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:26.007223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:26.007191 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:26.176376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:26.176352 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:26.486965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:26.486938 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:49:26.491655 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:49:26.491627 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2ba3211_cf44_412e_afad_12b8f5c792bd.slice/crio-b2319bd630e9d2f9cd4503308a4b9d602422f6c0d46ee4d7523c076dbfb021d5 WatchSource:0}: Error finding container b2319bd630e9d2f9cd4503308a4b9d602422f6c0d46ee4d7523c076dbfb021d5: Status 404 returned error can't find the container with id b2319bd630e9d2f9cd4503308a4b9d602422f6c0d46ee4d7523c076dbfb021d5 Oct 30 18:49:27.421514 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:27.421478 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"a2ba3211-cf44-412e-afad-12b8f5c792bd","Type":"ContainerStarted","Data":"e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d"} Oct 30 18:49:27.421957 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:27.421525 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:49:27.421957 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:27.421536 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"a2ba3211-cf44-412e-afad-12b8f5c792bd","Type":"ContainerStarted","Data":"b2319bd630e9d2f9cd4503308a4b9d602422f6c0d46ee4d7523c076dbfb021d5"} Oct 30 18:49:27.465017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:27.464970 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podStartSLOduration=11.464956828 podStartE2EDuration="11.464956828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:27.463868239 +0000 UTC m=+3180.452208749" watchObservedRunningTime="2026-10-30 18:49:27.464956828 +0000 UTC m=+3180.453297315" Oct 30 18:49:27.810022 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:27.809992 5014 scope.go:122] "RemoveContainer" containerID="376d9b1c7312a2978eda4425bff513dd6695f8f0fd2efec01d4fcffcf49796e8" Oct 30 18:49:42.401974 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:42.401942 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:49:43.334131 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:49:43.334097 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:49:48.431793 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:49:48.431765 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:50:13.332504 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:50:13.332468 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:50:43.332463 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:50:43.332427 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:50:51.925484 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:51.925453 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd"] Oct 30 18:50:51.928911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:51.928892 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:51.955709 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:51.955676 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 30 18:50:51.955823 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:51.955800 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 30 18:50:51.955933 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:51.955912 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-rl5v4\"" Oct 30 18:50:51.957527 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:51.957506 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd"] Oct 30 18:50:52.004005 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.003977 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5bc884688f-s42zd\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:52.004123 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.004021 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xhbd\" (UniqueName: \"kubernetes.io/projected/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-kube-api-access-4xhbd\") pod \"data-science-usage-logs-collector-5bc884688f-s42zd\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:52.004123 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.004064 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otc-internal\") pod \"data-science-usage-logs-collector-5bc884688f-s42zd\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:52.104943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.104918 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5bc884688f-s42zd\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:52.105038 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.104952 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xhbd\" (UniqueName: \"kubernetes.io/projected/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-kube-api-access-4xhbd\") pod \"data-science-usage-logs-collector-5bc884688f-s42zd\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:52.105038 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.104979 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otc-internal\") pod \"data-science-usage-logs-collector-5bc884688f-s42zd\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:52.105693 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.105669 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otc-internal\") pod \"data-science-usage-logs-collector-5bc884688f-s42zd\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:52.107482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.107460 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5bc884688f-s42zd\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:52.127797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.127775 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xhbd\" (UniqueName: \"kubernetes.io/projected/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-kube-api-access-4xhbd\") pod \"data-science-usage-logs-collector-5bc884688f-s42zd\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:52.240661 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.240586 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:52.538800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.538724 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd"] Oct 30 18:50:52.543325 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:50:52.543299 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0159ae_c287_4cb8_9bd2_f555ca8a7b74.slice/crio-52301c3e8cab5d0e38eff5b4bc8d8f3295e751ae6f7e51c87a640f0c389d99e6 WatchSource:0}: Error finding container 52301c3e8cab5d0e38eff5b4bc8d8f3295e751ae6f7e51c87a640f0c389d99e6: Status 404 returned error can't find the container with id 52301c3e8cab5d0e38eff5b4bc8d8f3295e751ae6f7e51c87a640f0c389d99e6 Oct 30 18:50:52.545266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.545235 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:52.545364 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.545289 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:52.801328 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.801254 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" event={"ID":"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74","Type":"ContainerStarted","Data":"49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697"} Oct 30 18:50:52.801328 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.801301 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" event={"ID":"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74","Type":"ContainerStarted","Data":"52301c3e8cab5d0e38eff5b4bc8d8f3295e751ae6f7e51c87a640f0c389d99e6"} Oct 30 18:50:52.842547 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:52.842500 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" podStartSLOduration=1.842483512 podStartE2EDuration="1.842483512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:52.842123869 +0000 UTC m=+3265.830464356" watchObservedRunningTime="2026-10-30 18:50:52.842483512 +0000 UTC m=+3265.830824000" Oct 30 18:50:57.209815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.209788 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8"] Oct 30 18:50:57.213473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.213450 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:50:57.238444 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.238421 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8"] Oct 30 18:50:57.242313 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.242284 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/09bfdccc-3d3f-49aa-887f-acbe706a0528-otc-internal\") pod \"data-science-usage-logs-collector-57868947f6-r82b8\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:50:57.242426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.242382 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjpj\" (UniqueName: \"kubernetes.io/projected/09bfdccc-3d3f-49aa-887f-acbe706a0528-kube-api-access-qpjpj\") pod \"data-science-usage-logs-collector-57868947f6-r82b8\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:50:57.242530 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.242504 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/09bfdccc-3d3f-49aa-887f-acbe706a0528-otlp-tls-certs\") pod \"data-science-usage-logs-collector-57868947f6-r82b8\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:50:57.343058 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.343032 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/09bfdccc-3d3f-49aa-887f-acbe706a0528-otlp-tls-certs\") pod \"data-science-usage-logs-collector-57868947f6-r82b8\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:50:57.343196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.343084 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/09bfdccc-3d3f-49aa-887f-acbe706a0528-otc-internal\") pod \"data-science-usage-logs-collector-57868947f6-r82b8\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:50:57.343196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.343112 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjpj\" (UniqueName: \"kubernetes.io/projected/09bfdccc-3d3f-49aa-887f-acbe706a0528-kube-api-access-qpjpj\") pod \"data-science-usage-logs-collector-57868947f6-r82b8\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:50:57.343893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.343808 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/09bfdccc-3d3f-49aa-887f-acbe706a0528-otc-internal\") pod \"data-science-usage-logs-collector-57868947f6-r82b8\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:50:57.345715 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.345695 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/09bfdccc-3d3f-49aa-887f-acbe706a0528-otlp-tls-certs\") pod \"data-science-usage-logs-collector-57868947f6-r82b8\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:50:57.366261 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.366237 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjpj\" (UniqueName: \"kubernetes.io/projected/09bfdccc-3d3f-49aa-887f-acbe706a0528-kube-api-access-qpjpj\") pod \"data-science-usage-logs-collector-57868947f6-r82b8\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:50:57.526691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.526628 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:50:57.832325 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.832295 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8"] Oct 30 18:50:57.837676 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:50:57.837637 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bfdccc_3d3f_49aa_887f_acbe706a0528.slice/crio-beebb1afa1b32417d2de5095eec7433a6afc4398adc8cc4d4ee383c118b9438e WatchSource:0}: Error finding container beebb1afa1b32417d2de5095eec7433a6afc4398adc8cc4d4ee383c118b9438e: Status 404 returned error can't find the container with id beebb1afa1b32417d2de5095eec7433a6afc4398adc8cc4d4ee383c118b9438e Oct 30 18:50:57.839843 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.839811 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:57.839936 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:57.839868 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:58.208761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.208667 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd"] Oct 30 18:50:58.212904 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.212879 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.229396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.228934 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\"" Oct 30 18:50:58.229396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.229074 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\"" Oct 30 18:50:58.229396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.229274 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 18:50:58.229755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.229411 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\"" Oct 30 18:50:58.229755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.229475 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\"" Oct 30 18:50:58.240369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.240344 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 18:50:58.242892 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.242869 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd"] Oct 30 18:50:58.250409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.250380 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tls-secret\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.250514 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.250435 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9jp7\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-kube-api-access-q9jp7\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.250514 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.250492 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.250664 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.250565 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.250664 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.250635 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-lokistack-gateway\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.250757 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.250691 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-rbac\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.250822 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.250799 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tenants\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.250861 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.250842 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.351351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.351322 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-rbac\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.351532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.351371 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tenants\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.351532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.351393 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.351532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.351441 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tls-secret\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.351532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.351473 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9jp7\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-kube-api-access-q9jp7\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.351532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.351498 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.351532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.351529 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.351903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.351713 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-lokistack-gateway\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.352324 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.352300 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-rbac\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.352668 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.352304 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.352797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.352701 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-lokistack-gateway\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.354665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.354636 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tenants\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.354756 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.354683 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tls-secret\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.354979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.354961 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.355296 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.355281 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.373829 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.373806 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9jp7\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-kube-api-access-q9jp7\") pod \"data-science-lokistack-gateway-6f9b9f4d87-bcgpd\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.529681 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.529606 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:58.830609 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.830561 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" event={"ID":"09bfdccc-3d3f-49aa-887f-acbe706a0528","Type":"ContainerStarted","Data":"c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017"} Oct 30 18:50:58.830778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.830632 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" event={"ID":"09bfdccc-3d3f-49aa-887f-acbe706a0528","Type":"ContainerStarted","Data":"beebb1afa1b32417d2de5095eec7433a6afc4398adc8cc4d4ee383c118b9438e"} Oct 30 18:50:58.857679 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.857640 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd"] Oct 30 18:50:58.861509 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:50:58.861481 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0d1316f_fbd8_48d6_b68a_420f873fdc96.slice/crio-7f3708181e76d4be2ae1757f3559d901adecb968ba99e8312a70cf049831d343 WatchSource:0}: Error finding container 7f3708181e76d4be2ae1757f3559d901adecb968ba99e8312a70cf049831d343: Status 404 returned error can't find the container with id 7f3708181e76d4be2ae1757f3559d901adecb968ba99e8312a70cf049831d343 Oct 30 18:50:58.871441 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.871399 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" podStartSLOduration=1.871386744 podStartE2EDuration="1.871386744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:58.869931097 +0000 UTC m=+3271.858271583" watchObservedRunningTime="2026-10-30 18:50:58.871386744 +0000 UTC m=+3271.859727231" Oct 30 18:50:58.901214 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.901186 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd"] Oct 30 18:50:58.901408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:58.901379 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" podUID="4d0159ae-c287-4cb8-9bd2-f555ca8a7b74" containerName="otc-container" containerID="cri-o://49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697" gracePeriod=30 Oct 30 18:50:59.159985 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.159953 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:59.261178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.261143 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-kube-api-access-4xhbd\" (UniqueName: \"kubernetes.io/projected/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-kube-api-access-4xhbd\") pod \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " Oct 30 18:50:59.261678 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.261204 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otlp-tls-certs\") pod \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " Oct 30 18:50:59.261678 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.261226 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otc-internal\") pod \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\" (UID: \"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74\") " Oct 30 18:50:59.261777 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.261720 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otc-internal" pod "4d0159ae-c287-4cb8-9bd2-f555ca8a7b74" (UID: "4d0159ae-c287-4cb8-9bd2-f555ca8a7b74"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:59.263540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.263512 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otlp-tls-certs" pod "4d0159ae-c287-4cb8-9bd2-f555ca8a7b74" (UID: "4d0159ae-c287-4cb8-9bd2-f555ca8a7b74"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:59.263738 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.263707 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-kube-api-access-4xhbd" pod "4d0159ae-c287-4cb8-9bd2-f555ca8a7b74" (UID: "4d0159ae-c287-4cb8-9bd2-f555ca8a7b74"). InnerVolumeSpecName "kube-api-access-4xhbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:59.362082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.362061 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xhbd\" (UniqueName: \"kubernetes.io/projected/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-kube-api-access-4xhbd\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:50:59.362200 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.362087 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:50:59.362200 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.362101 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:50:59.837226 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.837194 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" event={"ID":"d0d1316f-fbd8-48d6-b68a-420f873fdc96","Type":"ContainerStarted","Data":"17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104"} Oct 30 18:50:59.837413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.837246 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:59.837413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.837262 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" event={"ID":"d0d1316f-fbd8-48d6-b68a-420f873fdc96","Type":"ContainerStarted","Data":"3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3"} Oct 30 18:50:59.837413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.837276 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" event={"ID":"d0d1316f-fbd8-48d6-b68a-420f873fdc96","Type":"ContainerStarted","Data":"7f3708181e76d4be2ae1757f3559d901adecb968ba99e8312a70cf049831d343"} Oct 30 18:50:59.837413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.837385 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:59.838722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.838692 5014 generic.go:356] "Generic (PLEG): container finished" podID="4d0159ae-c287-4cb8-9bd2-f555ca8a7b74" containerID="49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697" exitCode=0 Oct 30 18:50:59.838863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.838741 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" Oct 30 18:50:59.838863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.838788 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" event={"ID":"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74","Type":"ContainerDied","Data":"49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697"} Oct 30 18:50:59.838863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.838833 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd" event={"ID":"4d0159ae-c287-4cb8-9bd2-f555ca8a7b74","Type":"ContainerDied","Data":"52301c3e8cab5d0e38eff5b4bc8d8f3295e751ae6f7e51c87a640f0c389d99e6"} Oct 30 18:50:59.838863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.838857 5014 scope.go:122] "RemoveContainer" containerID="49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697" Oct 30 18:50:59.845435 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.845413 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:59.847973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.847954 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:50:59.849766 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.849752 5014 scope.go:122] "RemoveContainer" containerID="49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697" Oct 30 18:50:59.850055 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:50:59.850036 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697\": container with ID starting with 49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697 not found: ID does not exist" containerID="49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697" Oct 30 18:50:59.850121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.850064 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697"} err="failed to get container status \"49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697\": rpc error: code = NotFound desc = could not find container \"49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697\": container with ID starting with 49e91ddb74142b89b889240e381302762d3fce7ea059eb59b483c9589740f697 not found: ID does not exist" Oct 30 18:50:59.881511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.881468 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" podStartSLOduration=1.881457731 podStartE2EDuration="1.881457731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:59.879851796 +0000 UTC m=+3272.868192283" watchObservedRunningTime="2026-10-30 18:50:59.881457731 +0000 UTC m=+3272.869798440" Oct 30 18:50:59.914229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.914208 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd"] Oct 30 18:50:59.919061 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:50:59.919042 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-s42zd"] Oct 30 18:51:00.064453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.064429 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 18:51:00.064831 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.064811 5014 cpu_manager.go:414] "containerMap: removing container" podUID="4d0159ae-c287-4cb8-9bd2-f555ca8a7b74" containerName="otc-container" Oct 30 18:51:00.064831 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.064831 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0159ae-c287-4cb8-9bd2-f555ca8a7b74" containerName="otc-container" Oct 30 18:51:00.065016 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.064941 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0159ae-c287-4cb8-9bd2-f555ca8a7b74" containerName="otc-container" Oct 30 18:51:00.069353 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.069337 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.083799 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.083779 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-logs-loki-secret\"" Oct 30 18:51:00.083946 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.083929 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-h78x9\"" Oct 30 18:51:00.083946 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.083940 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\"" Oct 30 18:51:00.084101 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.083952 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\"" Oct 30 18:51:00.084101 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.084057 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\"" Oct 30 18:51:00.091385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.091339 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 18:51:00.270119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.270088 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.270454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.270134 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-test-logs-loki-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.270454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.270155 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9qp\" (UniqueName: \"kubernetes.io/projected/060e797a-a209-4b93-be54-f3c3c72f6176-kube-api-access-9g9qp\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.270454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.270282 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.270454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.270331 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.270454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.270418 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.371324 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.371264 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-test-logs-loki-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.371324 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.371299 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9qp\" (UniqueName: \"kubernetes.io/projected/060e797a-a209-4b93-be54-f3c3c72f6176-kube-api-access-9g9qp\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.371522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.371337 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.371522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.371382 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.371522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.371414 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.371522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.371465 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.372183 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.372145 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.372350 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.372321 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.374049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.374028 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.374158 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.374120 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.374223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.374154 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-test-logs-loki-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:00.394119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:00.394097 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9qp\" (UniqueName: \"kubernetes.io/projected/060e797a-a209-4b93-be54-f3c3c72f6176-kube-api-access-9g9qp\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:01.141840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.141811 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0159ae-c287-4cb8-9bd2-f555ca8a7b74" path="/var/lib/kubelet/pods/4d0159ae-c287-4cb8-9bd2-f555ca8a7b74/volumes" Oct 30 18:51:01.830131 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.830098 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-5p2nr"] Oct 30 18:51:01.834817 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.834798 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:01.862464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.862440 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-5p2nr"] Oct 30 18:51:01.906418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.906391 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-metrics\"" Oct 30 18:51:01.906581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.906561 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"kube-root-ca.crt\"" Oct 30 18:51:01.906705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.906562 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"openshift-service-ca.crt\"" Oct 30 18:51:01.906771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.906715 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-dockercfg-hnvcg\"" Oct 30 18:51:01.918677 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.918655 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-config\"" Oct 30 18:51:01.918859 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.918846 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-token\"" Oct 30 18:51:01.930973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.930951 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-trustbundle\"" Oct 30 18:51:01.986058 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.986028 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-varlogpods\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:01.986174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.986075 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:01.986174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.986151 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:01.986274 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.986244 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23319930-78fb-4e56-9c7a-364c7ef6980c-tmp\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:01.986326 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.986274 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-entrypoint\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:01.986500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.986481 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-metrics\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:01.986651 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.986606 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:01.986742 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.986683 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpl65\" (UniqueName: \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-kube-api-access-lpl65\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:01.986816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.986798 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-trusted-ca\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:01.986874 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.986832 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-datadir\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:01.986938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:01.986883 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-sa-token\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.087482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087425 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-entrypoint\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.087482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087459 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-metrics\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.087685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087490 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.087685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087516 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpl65\" (UniqueName: \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-kube-api-access-lpl65\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.087685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087560 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-trusted-ca\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.087685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087580 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-datadir\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.087685 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:51:02.087588 5014 secret.go:189] Couldn't get secret openshift-logging/data-science-cluster-log-forwarder-metrics: secret "data-science-cluster-log-forwarder-metrics" not found Oct 30 18:51:02.087685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087636 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-sa-token\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.088058 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:51:02.087691 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-metrics podName:23319930-78fb-4e56-9c7a-364c7ef6980c nodeName:}" failed. No retries permitted until 2026-10-30 18:51:02.58766798 +0000 UTC m=+3275.576008467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics" (UniqueName: "kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-metrics") pod "data-science-cluster-log-forwarder-5p2nr" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c") : secret "data-science-cluster-log-forwarder-metrics" not found Oct 30 18:51:02.088058 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087723 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-varlogpods\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.088058 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087762 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.088058 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087764 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-datadir\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.088058 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087819 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-varlogpods\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.088058 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087832 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.088058 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.087882 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23319930-78fb-4e56-9c7a-364c7ef6980c-tmp\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.088420 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.088306 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-trusted-ca\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.088420 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.088341 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-entrypoint\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.088420 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.088376 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.088511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.088436 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.090106 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.090086 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23319930-78fb-4e56-9c7a-364c7ef6980c-tmp\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.090526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.090506 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.110470 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.110444 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpl65\" (UniqueName: \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-kube-api-access-lpl65\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.110549 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.110493 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-sa-token\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.207607 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.207561 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-5p2nr"] Oct 30 18:51:02.207822 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:51:02.207803 5014 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" podUID="23319930-78fb-4e56-9c7a-364c7ef6980c" Oct 30 18:51:02.600467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.600435 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-metrics\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.602915 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.602885 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-metrics\") pod \"data-science-cluster-log-forwarder-5p2nr\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.856220 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.856154 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:02.864447 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:02.864424 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:03.005228 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005206 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config\") pod \"23319930-78fb-4e56-9c7a-364c7ef6980c\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " Oct 30 18:51:03.005337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005239 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-trusted-ca\") pod \"23319930-78fb-4e56-9c7a-364c7ef6980c\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " Oct 30 18:51:03.005337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005265 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-metrics\" (UniqueName: \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-metrics\") pod \"23319930-78fb-4e56-9c7a-364c7ef6980c\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " Oct 30 18:51:03.005337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005298 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23319930-78fb-4e56-9c7a-364c7ef6980c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/23319930-78fb-4e56-9c7a-364c7ef6980c-tmp\") pod \"23319930-78fb-4e56-9c7a-364c7ef6980c\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " Oct 30 18:51:03.005337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005325 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config-openshift-service-cacrt\") pod \"23319930-78fb-4e56-9c7a-364c7ef6980c\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " Oct 30 18:51:03.005581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005373 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-varlogpods\" (UniqueName: \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-varlogpods\") pod \"23319930-78fb-4e56-9c7a-364c7ef6980c\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " Oct 30 18:51:03.005581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005402 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-data-science-cluster-log-forwarder-collector-token\") pod \"23319930-78fb-4e56-9c7a-364c7ef6980c\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " Oct 30 18:51:03.005581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005441 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-entrypoint\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-entrypoint\") pod \"23319930-78fb-4e56-9c7a-364c7ef6980c\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " Oct 30 18:51:03.005581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005470 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-datadir\" (UniqueName: \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-datadir\") pod \"23319930-78fb-4e56-9c7a-364c7ef6980c\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " Oct 30 18:51:03.005581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005502 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-kube-api-access-lpl65\" (UniqueName: \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-kube-api-access-lpl65\") pod \"23319930-78fb-4e56-9c7a-364c7ef6980c\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " Oct 30 18:51:03.005581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005548 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-sa-token\" (UniqueName: \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-sa-token\") pod \"23319930-78fb-4e56-9c7a-364c7ef6980c\" (UID: \"23319930-78fb-4e56-9c7a-364c7ef6980c\") " Oct 30 18:51:03.005581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005550 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-varlogpods" pod "23319930-78fb-4e56-9c7a-364c7ef6980c" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 18:51:03.005998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005684 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-datadir" pod "23319930-78fb-4e56-9c7a-364c7ef6980c" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 18:51:03.005998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005697 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config" pod "23319930-78fb-4e56-9c7a-364c7ef6980c" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:03.005998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005717 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-trusted-ca" pod "23319930-78fb-4e56-9c7a-364c7ef6980c" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:03.005998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005789 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config-openshift-service-cacrt" pod "23319930-78fb-4e56-9c7a-364c7ef6980c" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:03.005998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005971 5014 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-varlogpods\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.005998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005986 5014 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/23319930-78fb-4e56-9c7a-364c7ef6980c-datadir\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.005998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.005998 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.005998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.006012 5014 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-trusted-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.005998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.006028 5014 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-config-openshift-service-cacrt\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.006696 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.006116 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-entrypoint" pod "23319930-78fb-4e56-9c7a-364c7ef6980c" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:03.007921 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.007895 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-metrics" pod "23319930-78fb-4e56-9c7a-364c7ef6980c" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:51:03.008023 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.007923 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-kube-api-access-lpl65" pod "23319930-78fb-4e56-9c7a-364c7ef6980c" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c"). InnerVolumeSpecName "kube-api-access-lpl65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:03.008023 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.007949 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-sa-token" pod "23319930-78fb-4e56-9c7a-364c7ef6980c" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:03.008443 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.008428 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23319930-78fb-4e56-9c7a-364c7ef6980c-tmp" pod "23319930-78fb-4e56-9c7a-364c7ef6980c" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:51:03.008496 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.008437 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-data-science-cluster-log-forwarder-collector-token" pod "23319930-78fb-4e56-9c7a-364c7ef6980c" (UID: "23319930-78fb-4e56-9c7a-364c7ef6980c"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:51:03.106475 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.106427 5014 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-sa-token\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.106475 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.106450 5014 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-metrics\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.106475 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.106458 5014 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23319930-78fb-4e56-9c7a-364c7ef6980c-tmp\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.106475 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.106466 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/23319930-78fb-4e56-9c7a-364c7ef6980c-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.106475 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.106475 5014 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/23319930-78fb-4e56-9c7a-364c7ef6980c-entrypoint\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.106710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.106485 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpl65\" (UniqueName: \"kubernetes.io/projected/23319930-78fb-4e56-9c7a-364c7ef6980c-kube-api-access-lpl65\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.861053 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.861019 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-5p2nr" Oct 30 18:51:03.951200 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.951168 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-5p2nr"] Oct 30 18:51:03.955894 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.955870 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-5p2nr"] Oct 30 18:51:03.971536 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.971510 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-q9lj7"] Oct 30 18:51:03.976396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.976379 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:03.991296 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.991279 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"openshift-service-ca.crt\"" Oct 30 18:51:03.991389 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.991324 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"kube-root-ca.crt\"" Oct 30 18:51:03.991638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:03.991620 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-config\"" Oct 30 18:51:04.003767 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.003744 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-dockercfg-hnvcg\"" Oct 30 18:51:04.003869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.003776 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-token\"" Oct 30 18:51:04.004039 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.004018 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-metrics\"" Oct 30 18:51:04.004116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.004098 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-trustbundle\"" Oct 30 18:51:04.005583 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.005565 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-q9lj7"] Oct 30 18:51:04.114558 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.114504 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-trusted-ca\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.114558 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.114530 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrq4w\" (UniqueName: \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-kube-api-access-qrq4w\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.114818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.114561 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-varlogpods\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.114818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.114728 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-sa-token\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.114818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.114790 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-entrypoint\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.114980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.114830 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.114980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.114859 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afbed7cc-89d6-4052-b8c9-b199e2038344-tmp\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.114980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.114924 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-metrics\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.114980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.114952 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-datadir\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.115127 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.115016 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.115127 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.115060 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.216494 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.216454 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-entrypoint\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.216778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.216547 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.216778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.216580 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afbed7cc-89d6-4052-b8c9-b199e2038344-tmp\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.216778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.216634 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-metrics\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.216778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.216653 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-datadir\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.216778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.216709 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.216778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.216774 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.217408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.216797 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-trusted-ca\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.217408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.216813 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrq4w\" (UniqueName: \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-kube-api-access-qrq4w\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.217408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.216848 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-varlogpods\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.217408 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.216893 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-sa-token\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.217661 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.217492 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-entrypoint\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.217661 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.217567 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-datadir\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.217910 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.217851 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-varlogpods\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.217910 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.217867 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.218203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.218179 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-trusted-ca\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.218374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.218247 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.219342 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.219304 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afbed7cc-89d6-4052-b8c9-b199e2038344-tmp\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.219546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.219528 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-metrics\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.219702 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.219683 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.239598 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.239571 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-sa-token\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.240466 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.240443 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrq4w\" (UniqueName: \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-kube-api-access-qrq4w\") pod \"data-science-cluster-log-forwarder-q9lj7\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.288520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.288494 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:51:04.603063 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.603023 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-q9lj7"] Oct 30 18:51:04.607041 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:51:04.607013 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbed7cc_89d6_4052_b8c9_b199e2038344.slice/crio-a43925e487edb082a36a73c914ff444f6199382f04dd44b15612a0caa55349ac WatchSource:0}: Error finding container a43925e487edb082a36a73c914ff444f6199382f04dd44b15612a0caa55349ac: Status 404 returned error can't find the container with id a43925e487edb082a36a73c914ff444f6199382f04dd44b15612a0caa55349ac Oct 30 18:51:04.866743 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:04.866674 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" event={"ID":"afbed7cc-89d6-4052-b8c9-b199e2038344","Type":"ContainerStarted","Data":"a43925e487edb082a36a73c914ff444f6199382f04dd44b15612a0caa55349ac"} Oct 30 18:51:05.141691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:05.141617 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23319930-78fb-4e56-9c7a-364c7ef6980c" path="/var/lib/kubelet/pods/23319930-78fb-4e56-9c7a-364c7ef6980c/volumes" Oct 30 18:51:07.542701 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:07.542608 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d5f86fcc-ef20-41cc-ba28-8bf915495109\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:07.571239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:07.571210 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-d5f86fcc-ef20-41cc-ba28-8bf915495109\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:07.643659 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:07.643622 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d5f86fcc-ef20-41cc-ba28-8bf915495109\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:07.643840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:07.643756 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d5f86fcc-ef20-41cc-ba28-8bf915495109\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:07.657657 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:07.657620 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d5f86fcc-ef20-41cc-ba28-8bf915495109\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") DevicePath \"csi-b563bf34a4e95482873885e3be4aa19c4c2f32e4ed2d980c99a92f5ab5c32af3\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:07.890128 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:07.890080 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d5f86fcc-ef20-41cc-ba28-8bf915495109\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0924c22c3e28d10da544b15779236b4d550fa7c3f9b488298d62a32dfc22437d/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:07.918133 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:07.918101 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-d5f86fcc-ef20-41cc-ba28-8bf915495109\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:08.181129 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:08.181104 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:08.484404 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:08.484300 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 18:51:08.488421 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:51:08.488386 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod060e797a_a209_4b93_be54_f3c3c72f6176.slice/crio-f713282b2d027486d80f6e1bfa6d3464d5fd81e30073a7389da93c536ca5ed87 WatchSource:0}: Error finding container f713282b2d027486d80f6e1bfa6d3464d5fd81e30073a7389da93c536ca5ed87: Status 404 returned error can't find the container with id f713282b2d027486d80f6e1bfa6d3464d5fd81e30073a7389da93c536ca5ed87 Oct 30 18:51:08.897030 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:08.896991 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"060e797a-a209-4b93-be54-f3c3c72f6176","Type":"ContainerStarted","Data":"d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9"} Oct 30 18:51:08.897489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:08.897040 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"060e797a-a209-4b93-be54-f3c3c72f6176","Type":"ContainerStarted","Data":"f713282b2d027486d80f6e1bfa6d3464d5fd81e30073a7389da93c536ca5ed87"} Oct 30 18:51:08.897489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:08.897062 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:08.898901 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:08.898866 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" event={"ID":"afbed7cc-89d6-4052-b8c9-b199e2038344","Type":"ContainerStarted","Data":"85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4"} Oct 30 18:51:08.954108 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:08.954067 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podStartSLOduration=10.954055818 podStartE2EDuration="10.954055818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:08.941423021 +0000 UTC m=+3281.929763508" watchObservedRunningTime="2026-10-30 18:51:08.954055818 +0000 UTC m=+3281.942396305" Oct 30 18:51:08.986425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:08.986375 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" podStartSLOduration=2.521685369 podStartE2EDuration="5.986363035s" totalImagesPullingTime="3.464677666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:08.985701994 +0000 UTC m=+3281.974042478" watchObservedRunningTime="2026-10-30 18:51:08.986363035 +0000 UTC m=+3281.974703523" Oct 30 18:51:13.333194 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:51:13.333158 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:51:27.449224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:27.449091 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:51:27.460982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:27.460962 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:51:29.908993 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:51:29.908968 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:51:43.338355 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:51:43.338307 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:52:13.332404 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:13.332370 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:52:21.678706 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:21.678672 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 18:52:21.679362 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:21.678875 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podUID="060e797a-a209-4b93-be54-f3c3c72f6176" containerName="loki-index-gateway" containerID="cri-o://d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9" gracePeriod=30 Oct 30 18:52:21.740339 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:21.740296 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd"] Oct 30 18:52:21.740976 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:21.740574 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" podUID="d0d1316f-fbd8-48d6-b68a-420f873fdc96" containerName="gateway" containerID="cri-o://3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3" gracePeriod=30 Oct 30 18:52:21.740976 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:21.740729 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" podUID="d0d1316f-fbd8-48d6-b68a-420f873fdc96" containerName="opa" containerID="cri-o://17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104" gracePeriod=30 Oct 30 18:52:21.834103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:21.834063 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-q9lj7"] Oct 30 18:52:21.834412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:21.834372 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" podUID="afbed7cc-89d6-4052-b8c9-b199e2038344" containerName="collector" containerID="cri-o://85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4" gracePeriod=10 Oct 30 18:52:21.994687 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:21.994658 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:52:22.073744 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.073707 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-ca-bundle\") pod \"060e797a-a209-4b93-be54-f3c3c72f6176\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " Oct 30 18:52:22.073915 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.073802 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-test-logs-loki-secret\") pod \"060e797a-a209-4b93-be54-f3c3c72f6176\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " Oct 30 18:52:22.073915 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.073853 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-http\") pod \"060e797a-a209-4b93-be54-f3c3c72f6176\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " Oct 30 18:52:22.073915 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.073887 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060e797a-a209-4b93-be54-f3c3c72f6176-kube-api-access-9g9qp\" (UniqueName: \"kubernetes.io/projected/060e797a-a209-4b93-be54-f3c3c72f6176-kube-api-access-9g9qp\") pod \"060e797a-a209-4b93-be54-f3c3c72f6176\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " Oct 30 18:52:22.074114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.073967 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-config\" (UniqueName: \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-config\") pod \"060e797a-a209-4b93-be54-f3c3c72f6176\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " Oct 30 18:52:22.074114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.074073 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2\") pod \"060e797a-a209-4b93-be54-f3c3c72f6176\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " Oct 30 18:52:22.074250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.074134 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-grpc\") pod \"060e797a-a209-4b93-be54-f3c3c72f6176\" (UID: \"060e797a-a209-4b93-be54-f3c3c72f6176\") " Oct 30 18:52:22.074250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.074142 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-ca-bundle" pod "060e797a-a209-4b93-be54-f3c3c72f6176" (UID: "060e797a-a209-4b93-be54-f3c3c72f6176"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.077374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.074454 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.077374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.074731 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-config" pod "060e797a-a209-4b93-be54-f3c3c72f6176" (UID: "060e797a-a209-4b93-be54-f3c3c72f6176"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.077374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.077294 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-grpc" pod "060e797a-a209-4b93-be54-f3c3c72f6176" (UID: "060e797a-a209-4b93-be54-f3c3c72f6176"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:22.077374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.077307 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-http" pod "060e797a-a209-4b93-be54-f3c3c72f6176" (UID: "060e797a-a209-4b93-be54-f3c3c72f6176"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:22.077374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.077320 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060e797a-a209-4b93-be54-f3c3c72f6176-kube-api-access-9g9qp" pod "060e797a-a209-4b93-be54-f3c3c72f6176" (UID: "060e797a-a209-4b93-be54-f3c3c72f6176"). InnerVolumeSpecName "kube-api-access-9g9qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:22.077374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.077351 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-test-logs-loki-secret" pod "060e797a-a209-4b93-be54-f3c3c72f6176" (UID: "060e797a-a209-4b93-be54-f3c3c72f6176"). InnerVolumeSpecName "test-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:22.078295 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.078277 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2" pod "060e797a-a209-4b93-be54-f3c3c72f6176" (UID: "060e797a-a209-4b93-be54-f3c3c72f6176"). InnerVolumeSpecName "pvc-d5f86fcc-ef20-41cc-ba28-8bf915495109". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:52:22.096127 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.096109 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:52:22.175176 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175143 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-varlogpods\" (UniqueName: \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-varlogpods\") pod \"afbed7cc-89d6-4052-b8c9-b199e2038344\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " Oct 30 18:52:22.175337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175198 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-data-science-cluster-log-forwarder-collector-token\") pod \"afbed7cc-89d6-4052-b8c9-b199e2038344\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " Oct 30 18:52:22.175337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175226 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-entrypoint\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-entrypoint\") pod \"afbed7cc-89d6-4052-b8c9-b199e2038344\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " Oct 30 18:52:22.175337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175241 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-datadir\" (UniqueName: \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-datadir\") pod \"afbed7cc-89d6-4052-b8c9-b199e2038344\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " Oct 30 18:52:22.175337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175247 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-varlogpods" pod "afbed7cc-89d6-4052-b8c9-b199e2038344" (UID: "afbed7cc-89d6-4052-b8c9-b199e2038344"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 18:52:22.175337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175271 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config-openshift-service-cacrt\") pod \"afbed7cc-89d6-4052-b8c9-b199e2038344\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " Oct 30 18:52:22.175337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175307 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-datadir" pod "afbed7cc-89d6-4052-b8c9-b199e2038344" (UID: "afbed7cc-89d6-4052-b8c9-b199e2038344"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 18:52:22.175632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175355 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-sa-token\" (UniqueName: \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-sa-token\") pod \"afbed7cc-89d6-4052-b8c9-b199e2038344\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " Oct 30 18:52:22.175632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175387 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-trusted-ca\") pod \"afbed7cc-89d6-4052-b8c9-b199e2038344\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " Oct 30 18:52:22.175632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175418 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-metrics\" (UniqueName: \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-metrics\") pod \"afbed7cc-89d6-4052-b8c9-b199e2038344\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " Oct 30 18:52:22.175632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175479 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config\") pod \"afbed7cc-89d6-4052-b8c9-b199e2038344\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " Oct 30 18:52:22.175632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175521 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbed7cc-89d6-4052-b8c9-b199e2038344-tmp\" (UniqueName: \"kubernetes.io/empty-dir/afbed7cc-89d6-4052-b8c9-b199e2038344-tmp\") pod \"afbed7cc-89d6-4052-b8c9-b199e2038344\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " Oct 30 18:52:22.175632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175567 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-kube-api-access-qrq4w\" (UniqueName: \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-kube-api-access-qrq4w\") pod \"afbed7cc-89d6-4052-b8c9-b199e2038344\" (UID: \"afbed7cc-89d6-4052-b8c9-b199e2038344\") " Oct 30 18:52:22.175953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175772 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config-openshift-service-cacrt" pod "afbed7cc-89d6-4052-b8c9-b199e2038344" (UID: "afbed7cc-89d6-4052-b8c9-b199e2038344"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.175953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.175865 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-trusted-ca" pod "afbed7cc-89d6-4052-b8c9-b199e2038344" (UID: "afbed7cc-89d6-4052-b8c9-b199e2038344"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.176083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.176057 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/060e797a-a209-4b93-be54-f3c3c72f6176-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.176144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.176088 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config" pod "afbed7cc-89d6-4052-b8c9-b199e2038344" (UID: "afbed7cc-89d6-4052-b8c9-b199e2038344"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.176144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.176099 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d5f86fcc-ef20-41cc-ba28-8bf915495109\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:52:22.176144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.176118 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.176144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.176133 5014 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-varlogpods\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.176375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.176147 5014 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/afbed7cc-89d6-4052-b8c9-b199e2038344-datadir\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.176375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.176161 5014 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config-openshift-service-cacrt\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.176375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.176174 5014 reconciler_common.go:299] "Volume detached for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-test-logs-loki-secret\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.176375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.176188 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/060e797a-a209-4b93-be54-f3c3c72f6176-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.176375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.176209 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g9qp\" (UniqueName: \"kubernetes.io/projected/060e797a-a209-4b93-be54-f3c3c72f6176-kube-api-access-9g9qp\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.176375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.176226 5014 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-trusted-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.178155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.178129 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-sa-token" pod "afbed7cc-89d6-4052-b8c9-b199e2038344" (UID: "afbed7cc-89d6-4052-b8c9-b199e2038344"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:22.178406 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.178381 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-data-science-cluster-log-forwarder-collector-token" pod "afbed7cc-89d6-4052-b8c9-b199e2038344" (UID: "afbed7cc-89d6-4052-b8c9-b199e2038344"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:22.178703 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.178682 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbed7cc-89d6-4052-b8c9-b199e2038344-tmp" pod "afbed7cc-89d6-4052-b8c9-b199e2038344" (UID: "afbed7cc-89d6-4052-b8c9-b199e2038344"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:52:22.179252 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.179229 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-kube-api-access-qrq4w" pod "afbed7cc-89d6-4052-b8c9-b199e2038344" (UID: "afbed7cc-89d6-4052-b8c9-b199e2038344"). InnerVolumeSpecName "kube-api-access-qrq4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:22.179712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.179689 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-metrics" pod "afbed7cc-89d6-4052-b8c9-b199e2038344" (UID: "afbed7cc-89d6-4052-b8c9-b199e2038344"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:22.183076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.183051 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-entrypoint" pod "afbed7cc-89d6-4052-b8c9-b199e2038344" (UID: "afbed7cc-89d6-4052-b8c9-b199e2038344"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.189640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.189626 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-d5f86fcc-ef20-41cc-ba28-8bf915495109" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:52:22.256560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.256479 5014 generic.go:356] "Generic (PLEG): container finished" podID="afbed7cc-89d6-4052-b8c9-b199e2038344" containerID="85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4" exitCode=0 Oct 30 18:52:22.256763 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.256580 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" Oct 30 18:52:22.256763 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.256657 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" event={"ID":"afbed7cc-89d6-4052-b8c9-b199e2038344","Type":"ContainerDied","Data":"85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4"} Oct 30 18:52:22.256763 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.256719 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-q9lj7" event={"ID":"afbed7cc-89d6-4052-b8c9-b199e2038344","Type":"ContainerDied","Data":"a43925e487edb082a36a73c914ff444f6199382f04dd44b15612a0caa55349ac"} Oct 30 18:52:22.256763 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.256752 5014 scope.go:122] "RemoveContainer" containerID="85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4" Oct 30 18:52:22.259340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.259316 5014 generic.go:356] "Generic (PLEG): container finished" podID="d0d1316f-fbd8-48d6-b68a-420f873fdc96" containerID="17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104" exitCode=0 Oct 30 18:52:22.259480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.259348 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" event={"ID":"d0d1316f-fbd8-48d6-b68a-420f873fdc96","Type":"ContainerDied","Data":"17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104"} Oct 30 18:52:22.261022 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.260919 5014 generic.go:356] "Generic (PLEG): container finished" podID="060e797a-a209-4b93-be54-f3c3c72f6176" containerID="d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9" exitCode=0 Oct 30 18:52:22.261022 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.260958 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 18:52:22.261022 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.260960 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"060e797a-a209-4b93-be54-f3c3c72f6176","Type":"ContainerDied","Data":"d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9"} Oct 30 18:52:22.261022 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.260999 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"060e797a-a209-4b93-be54-f3c3c72f6176","Type":"ContainerDied","Data":"f713282b2d027486d80f6e1bfa6d3464d5fd81e30073a7389da93c536ca5ed87"} Oct 30 18:52:22.273137 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.273114 5014 scope.go:122] "RemoveContainer" containerID="85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4" Oct 30 18:52:22.273490 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:22.273458 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4\": container with ID starting with 85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4 not found: ID does not exist" containerID="85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4" Oct 30 18:52:22.273605 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.273489 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4"} err="failed to get container status \"85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4\": rpc error: code = NotFound desc = could not find container \"85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4\": container with ID starting with 85dc883071e281471c783ee5cd15ac77f71eba0fc19dd67e1c119b8f14700db4 not found: ID does not exist" Oct 30 18:52:22.273605 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.273517 5014 scope.go:122] "RemoveContainer" containerID="d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9" Oct 30 18:52:22.276643 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.276623 5014 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-entrypoint\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.276734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.276648 5014 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-sa-token\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.276734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.276659 5014 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-metrics\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.276734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.276670 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-d5f86fcc-ef20-41cc-ba28-8bf915495109\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00b4afd2b295abfd2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-b563bf34a4e95482873885e3be4aa19c4c2f32e4ed2d980c99a92f5ab5c32af3\"" Oct 30 18:52:22.276734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.276680 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afbed7cc-89d6-4052-b8c9-b199e2038344-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.276734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.276694 5014 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afbed7cc-89d6-4052-b8c9-b199e2038344-tmp\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.276734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.276703 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrq4w\" (UniqueName: \"kubernetes.io/projected/afbed7cc-89d6-4052-b8c9-b199e2038344-kube-api-access-qrq4w\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.276734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.276712 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/afbed7cc-89d6-4052-b8c9-b199e2038344-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.282963 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.282939 5014 scope.go:122] "RemoveContainer" containerID="d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9" Oct 30 18:52:22.283254 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:22.283235 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9\": container with ID starting with d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9 not found: ID does not exist" containerID="d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9" Oct 30 18:52:22.283317 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.283259 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9"} err="failed to get container status \"d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9\": rpc error: code = NotFound desc = could not find container \"d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9\": container with ID starting with d7f0fabf7cd748e692326040c90dbd0716d868bcd4bdc7a265120b08b21800c9 not found: ID does not exist" Oct 30 18:52:22.324644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.324617 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 18:52:22.330496 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.330472 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 18:52:22.364420 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.364387 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-q9lj7"] Oct 30 18:52:22.368616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.368576 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-q9lj7"] Oct 30 18:52:22.592661 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.592630 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6"] Oct 30 18:52:22.593004 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.592990 5014 cpu_manager.go:414] "containerMap: removing container" podUID="060e797a-a209-4b93-be54-f3c3c72f6176" containerName="loki-index-gateway" Oct 30 18:52:22.593004 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.593003 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060e797a-a209-4b93-be54-f3c3c72f6176" containerName="loki-index-gateway" Oct 30 18:52:22.593126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.593027 5014 cpu_manager.go:414] "containerMap: removing container" podUID="afbed7cc-89d6-4052-b8c9-b199e2038344" containerName="collector" Oct 30 18:52:22.593126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.593032 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afbed7cc-89d6-4052-b8c9-b199e2038344" containerName="collector" Oct 30 18:52:22.593126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.593104 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="afbed7cc-89d6-4052-b8c9-b199e2038344" containerName="collector" Oct 30 18:52:22.593126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.593122 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="060e797a-a209-4b93-be54-f3c3c72f6176" containerName="loki-index-gateway" Oct 30 18:52:22.598070 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.598045 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:22.616713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.616691 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6"] Oct 30 18:52:22.680288 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.680260 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/92e92ebc-c270-46a1-b5fa-11223593671f-otlp-tls-certs\") pod \"data-science-usage-logs-collector-6b8fc67d75-76qc6\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:22.680687 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.680309 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s287n\" (UniqueName: \"kubernetes.io/projected/92e92ebc-c270-46a1-b5fa-11223593671f-kube-api-access-s287n\") pod \"data-science-usage-logs-collector-6b8fc67d75-76qc6\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:22.680687 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.680336 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/92e92ebc-c270-46a1-b5fa-11223593671f-otc-internal\") pod \"data-science-usage-logs-collector-6b8fc67d75-76qc6\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:22.781754 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.781716 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/92e92ebc-c270-46a1-b5fa-11223593671f-otlp-tls-certs\") pod \"data-science-usage-logs-collector-6b8fc67d75-76qc6\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:22.781903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.781828 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s287n\" (UniqueName: \"kubernetes.io/projected/92e92ebc-c270-46a1-b5fa-11223593671f-kube-api-access-s287n\") pod \"data-science-usage-logs-collector-6b8fc67d75-76qc6\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:22.781903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.781859 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/92e92ebc-c270-46a1-b5fa-11223593671f-otc-internal\") pod \"data-science-usage-logs-collector-6b8fc67d75-76qc6\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:22.782527 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.782502 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/92e92ebc-c270-46a1-b5fa-11223593671f-otc-internal\") pod \"data-science-usage-logs-collector-6b8fc67d75-76qc6\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:22.784314 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.784294 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/92e92ebc-c270-46a1-b5fa-11223593671f-otlp-tls-certs\") pod \"data-science-usage-logs-collector-6b8fc67d75-76qc6\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:22.805180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.805155 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s287n\" (UniqueName: \"kubernetes.io/projected/92e92ebc-c270-46a1-b5fa-11223593671f-kube-api-access-s287n\") pod \"data-science-usage-logs-collector-6b8fc67d75-76qc6\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:22.910138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:22.910105 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:23.001099 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.001072 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:52:23.085839 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.085812 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-kube-api-access-q9jp7\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-kube-api-access-q9jp7\") pod \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " Oct 30 18:52:23.085982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.085870 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-client-http\") pod \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " Oct 30 18:52:23.085982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.085913 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tls-secret\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tls-secret\") pod \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " Oct 30 18:52:23.085982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.085943 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-ca-bundle\") pod \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " Oct 30 18:52:23.086266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.086006 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tenants\" (UniqueName: \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tenants\") pod \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " Oct 30 18:52:23.086266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.086045 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-lokistack-gateway\") pod \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " Oct 30 18:52:23.086266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.086108 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-ca-bundle\") pod \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " Oct 30 18:52:23.086266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.086166 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-rbac\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-rbac\") pod \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\" (UID: \"d0d1316f-fbd8-48d6-b68a-420f873fdc96\") " Oct 30 18:52:23.086566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.086518 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-ca-bundle" pod "d0d1316f-fbd8-48d6-b68a-420f873fdc96" (UID: "d0d1316f-fbd8-48d6-b68a-420f873fdc96"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:23.087218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.087194 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.088434 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.088411 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tls-secret" pod "d0d1316f-fbd8-48d6-b68a-420f873fdc96" (UID: "d0d1316f-fbd8-48d6-b68a-420f873fdc96"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:23.088877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.088842 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-client-http" pod "d0d1316f-fbd8-48d6-b68a-420f873fdc96" (UID: "d0d1316f-fbd8-48d6-b68a-420f873fdc96"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:23.088877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.088855 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-kube-api-access-q9jp7" pod "d0d1316f-fbd8-48d6-b68a-420f873fdc96" (UID: "d0d1316f-fbd8-48d6-b68a-420f873fdc96"). InnerVolumeSpecName "kube-api-access-q9jp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:23.088877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.088871 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-ca-bundle" pod "d0d1316f-fbd8-48d6-b68a-420f873fdc96" (UID: "d0d1316f-fbd8-48d6-b68a-420f873fdc96"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:23.093398 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.093377 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tenants" pod "d0d1316f-fbd8-48d6-b68a-420f873fdc96" (UID: "d0d1316f-fbd8-48d6-b68a-420f873fdc96"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:23.093574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.093556 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-rbac" pod "d0d1316f-fbd8-48d6-b68a-420f873fdc96" (UID: "d0d1316f-fbd8-48d6-b68a-420f873fdc96"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:23.093653 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.093620 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-lokistack-gateway" pod "d0d1316f-fbd8-48d6-b68a-420f873fdc96" (UID: "d0d1316f-fbd8-48d6-b68a-420f873fdc96"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:23.142526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.142474 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060e797a-a209-4b93-be54-f3c3c72f6176" path="/var/lib/kubelet/pods/060e797a-a209-4b93-be54-f3c3c72f6176/volumes" Oct 30 18:52:23.142941 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.142925 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afbed7cc-89d6-4052-b8c9-b199e2038344" path="/var/lib/kubelet/pods/afbed7cc-89d6-4052-b8c9-b199e2038344/volumes" Oct 30 18:52:23.188345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.188319 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.188345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.188346 5014 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tls-secret\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.188467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.188356 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.188467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.188365 5014 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/d0d1316f-fbd8-48d6-b68a-420f873fdc96-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.188467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.188375 5014 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-lokistack-gateway\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.188467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.188384 5014 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d0d1316f-fbd8-48d6-b68a-420f873fdc96-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.188467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.188392 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9jp7\" (UniqueName: \"kubernetes.io/projected/d0d1316f-fbd8-48d6-b68a-420f873fdc96-kube-api-access-q9jp7\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.217540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.217517 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6"] Oct 30 18:52:23.222161 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:52:23.222137 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e92ebc_c270_46a1_b5fa_11223593671f.slice/crio-31e8f0f17c86574d624df02a112e97422c2da539948e043208617d2b2bc8f64c WatchSource:0}: Error finding container 31e8f0f17c86574d624df02a112e97422c2da539948e043208617d2b2bc8f64c: Status 404 returned error can't find the container with id 31e8f0f17c86574d624df02a112e97422c2da539948e043208617d2b2bc8f64c Oct 30 18:52:23.224165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.224137 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:23.224247 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.224190 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:23.268224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.268195 5014 generic.go:356] "Generic (PLEG): container finished" podID="d0d1316f-fbd8-48d6-b68a-420f873fdc96" containerID="3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3" exitCode=0 Oct 30 18:52:23.268337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.268260 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" Oct 30 18:52:23.268337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.268268 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" event={"ID":"d0d1316f-fbd8-48d6-b68a-420f873fdc96","Type":"ContainerDied","Data":"3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3"} Oct 30 18:52:23.268337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.268307 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd" event={"ID":"d0d1316f-fbd8-48d6-b68a-420f873fdc96","Type":"ContainerDied","Data":"7f3708181e76d4be2ae1757f3559d901adecb968ba99e8312a70cf049831d343"} Oct 30 18:52:23.268337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.268330 5014 scope.go:122] "RemoveContainer" containerID="17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104" Oct 30 18:52:23.272231 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.272202 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" event={"ID":"92e92ebc-c270-46a1-b5fa-11223593671f","Type":"ContainerStarted","Data":"31e8f0f17c86574d624df02a112e97422c2da539948e043208617d2b2bc8f64c"} Oct 30 18:52:23.277929 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.277905 5014 scope.go:122] "RemoveContainer" containerID="3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3" Oct 30 18:52:23.286220 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.286165 5014 scope.go:122] "RemoveContainer" containerID="17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104" Oct 30 18:52:23.286530 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:23.286451 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104\": container with ID starting with 17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104 not found: ID does not exist" containerID="17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104" Oct 30 18:52:23.286530 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.286482 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104"} err="failed to get container status \"17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104\": rpc error: code = NotFound desc = could not find container \"17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104\": container with ID starting with 17d4480c93e47dc7deeb4ef234ad3193e47b162d44cf2a283b2babf06bca8104 not found: ID does not exist" Oct 30 18:52:23.286530 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.286504 5014 scope.go:122] "RemoveContainer" containerID="3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3" Oct 30 18:52:23.286892 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:23.286871 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3\": container with ID starting with 3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3 not found: ID does not exist" containerID="3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3" Oct 30 18:52:23.286957 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.286900 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3"} err="failed to get container status \"3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3\": rpc error: code = NotFound desc = could not find container \"3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3\": container with ID starting with 3eed48628b6e73d3517c5c5e8a8cbcebb15dd8fc2814b5f44ed7a6441ccb89b3 not found: ID does not exist" Oct 30 18:52:23.320372 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.320337 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd"] Oct 30 18:52:23.326774 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:23.326746 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-bcgpd"] Oct 30 18:52:24.279420 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.279388 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" event={"ID":"92e92ebc-c270-46a1-b5fa-11223593671f","Type":"ContainerStarted","Data":"497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77"} Oct 30 18:52:24.320096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.320052 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" podStartSLOduration=2.320038981 podStartE2EDuration="2.320038981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:24.319362133 +0000 UTC m=+3357.307702621" watchObservedRunningTime="2026-10-30 18:52:24.320038981 +0000 UTC m=+3357.308379461" Oct 30 18:52:24.351338 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.351311 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8"] Oct 30 18:52:24.351565 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.351537 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" podUID="09bfdccc-3d3f-49aa-887f-acbe706a0528" containerName="otc-container" containerID="cri-o://c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017" gracePeriod=30 Oct 30 18:52:24.395737 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:24.395711 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found Oct 30 18:52:24.395883 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:24.395788 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e92ebc-c270-46a1-b5fa-11223593671f-otlp-tls-certs podName:92e92ebc-c270-46a1-b5fa-11223593671f nodeName:}" failed. No retries permitted until 2026-10-30 18:52:24.895766745 +0000 UTC m=+3357.884107233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/92e92ebc-c270-46a1-b5fa-11223593671f-otlp-tls-certs") pod "data-science-usage-logs-collector-6b8fc67d75-76qc6" (UID: "92e92ebc-c270-46a1-b5fa-11223593671f") : secret "data-science-usage-logs-collector-headless-tls" not found Oct 30 18:52:24.624096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.624071 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:52:24.699362 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.699324 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09bfdccc-3d3f-49aa-887f-acbe706a0528-otc-internal\" (UniqueName: \"kubernetes.io/configmap/09bfdccc-3d3f-49aa-887f-acbe706a0528-otc-internal\") pod \"09bfdccc-3d3f-49aa-887f-acbe706a0528\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " Oct 30 18:52:24.699476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.699406 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09bfdccc-3d3f-49aa-887f-acbe706a0528-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/09bfdccc-3d3f-49aa-887f-acbe706a0528-otlp-tls-certs\") pod \"09bfdccc-3d3f-49aa-887f-acbe706a0528\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " Oct 30 18:52:24.699545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.699527 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bfdccc-3d3f-49aa-887f-acbe706a0528-kube-api-access-qpjpj\" (UniqueName: \"kubernetes.io/projected/09bfdccc-3d3f-49aa-887f-acbe706a0528-kube-api-access-qpjpj\") pod \"09bfdccc-3d3f-49aa-887f-acbe706a0528\" (UID: \"09bfdccc-3d3f-49aa-887f-acbe706a0528\") " Oct 30 18:52:24.699749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.699726 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bfdccc-3d3f-49aa-887f-acbe706a0528-otc-internal" pod "09bfdccc-3d3f-49aa-887f-acbe706a0528" (UID: "09bfdccc-3d3f-49aa-887f-acbe706a0528"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:24.699865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.699844 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/09bfdccc-3d3f-49aa-887f-acbe706a0528-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:24.701611 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.701571 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09bfdccc-3d3f-49aa-887f-acbe706a0528-otlp-tls-certs" pod "09bfdccc-3d3f-49aa-887f-acbe706a0528" (UID: "09bfdccc-3d3f-49aa-887f-acbe706a0528"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:24.701775 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.701752 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bfdccc-3d3f-49aa-887f-acbe706a0528-kube-api-access-qpjpj" pod "09bfdccc-3d3f-49aa-887f-acbe706a0528" (UID: "09bfdccc-3d3f-49aa-887f-acbe706a0528"). InnerVolumeSpecName "kube-api-access-qpjpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:24.800893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.800843 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpjpj\" (UniqueName: \"kubernetes.io/projected/09bfdccc-3d3f-49aa-887f-acbe706a0528-kube-api-access-qpjpj\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:24.800893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:24.800863 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/09bfdccc-3d3f-49aa-887f-acbe706a0528-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:25.141168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:25.141141 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d1316f-fbd8-48d6-b68a-420f873fdc96" path="/var/lib/kubelet/pods/d0d1316f-fbd8-48d6-b68a-420f873fdc96/volumes" Oct 30 18:52:25.285765 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:25.285732 5014 generic.go:356] "Generic (PLEG): container finished" podID="09bfdccc-3d3f-49aa-887f-acbe706a0528" containerID="c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017" exitCode=0 Oct 30 18:52:25.286197 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:25.285781 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" Oct 30 18:52:25.286197 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:25.285824 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" event={"ID":"09bfdccc-3d3f-49aa-887f-acbe706a0528","Type":"ContainerDied","Data":"c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017"} Oct 30 18:52:25.286197 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:25.285893 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8" event={"ID":"09bfdccc-3d3f-49aa-887f-acbe706a0528","Type":"ContainerDied","Data":"beebb1afa1b32417d2de5095eec7433a6afc4398adc8cc4d4ee383c118b9438e"} Oct 30 18:52:25.286197 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:25.285932 5014 scope.go:122] "RemoveContainer" containerID="c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017" Oct 30 18:52:25.294838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:25.294822 5014 scope.go:122] "RemoveContainer" containerID="c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017" Oct 30 18:52:25.295107 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:25.295089 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017\": container with ID starting with c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017 not found: ID does not exist" containerID="c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017" Oct 30 18:52:25.295169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:25.295114 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017"} err="failed to get container status \"c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017\": rpc error: code = NotFound desc = could not find container \"c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017\": container with ID starting with c228e556df8c19829e9467c77d8f46ed46356fc2582638a8e06372fe12438017 not found: ID does not exist" Oct 30 18:52:25.335812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:25.335793 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8"] Oct 30 18:52:25.340616 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:25.340570 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57868947f6-r82b8"] Oct 30 18:52:27.143685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:27.143655 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09bfdccc-3d3f-49aa-887f-acbe706a0528" path="/var/lib/kubelet/pods/09bfdccc-3d3f-49aa-887f-acbe706a0528/volumes" Oct 30 18:52:30.115492 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:30.115461 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:52:30.116075 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:30.115648 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podUID="a2ba3211-cf44-412e-afad-12b8f5c792bd" containerName="loki-compactor" containerID="cri-o://e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d" gracePeriod=30 Oct 30 18:52:31.092631 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.092607 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:31.151388 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.151311 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-http\") pod \"a2ba3211-cf44-412e-afad-12b8f5c792bd\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " Oct 30 18:52:31.151854 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.151417 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"a2ba3211-cf44-412e-afad-12b8f5c792bd\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " Oct 30 18:52:31.151854 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.151441 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-config\" (UniqueName: \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-config\") pod \"a2ba3211-cf44-412e-afad-12b8f5c792bd\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " Oct 30 18:52:31.151854 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.151462 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-grpc\") pod \"a2ba3211-cf44-412e-afad-12b8f5c792bd\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " Oct 30 18:52:31.151854 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.151503 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-test-usage-logs-loki-secret\") pod \"a2ba3211-cf44-412e-afad-12b8f5c792bd\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " Oct 30 18:52:31.151854 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.151555 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-ca-bundle\") pod \"a2ba3211-cf44-412e-afad-12b8f5c792bd\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " Oct 30 18:52:31.151854 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.151584 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ba3211-cf44-412e-afad-12b8f5c792bd-kube-api-access-frm6g\" (UniqueName: \"kubernetes.io/projected/a2ba3211-cf44-412e-afad-12b8f5c792bd-kube-api-access-frm6g\") pod \"a2ba3211-cf44-412e-afad-12b8f5c792bd\" (UID: \"a2ba3211-cf44-412e-afad-12b8f5c792bd\") " Oct 30 18:52:31.152390 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.152362 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-config" pod "a2ba3211-cf44-412e-afad-12b8f5c792bd" (UID: "a2ba3211-cf44-412e-afad-12b8f5c792bd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:31.152540 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.152520 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-ca-bundle" pod "a2ba3211-cf44-412e-afad-12b8f5c792bd" (UID: "a2ba3211-cf44-412e-afad-12b8f5c792bd"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:31.154354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.154320 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-grpc" pod "a2ba3211-cf44-412e-afad-12b8f5c792bd" (UID: "a2ba3211-cf44-412e-afad-12b8f5c792bd"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:31.154456 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.154363 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-test-usage-logs-loki-secret" pod "a2ba3211-cf44-412e-afad-12b8f5c792bd" (UID: "a2ba3211-cf44-412e-afad-12b8f5c792bd"). InnerVolumeSpecName "test-usage-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:31.154456 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.154418 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-http" pod "a2ba3211-cf44-412e-afad-12b8f5c792bd" (UID: "a2ba3211-cf44-412e-afad-12b8f5c792bd"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:31.154907 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.154887 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879" pod "a2ba3211-cf44-412e-afad-12b8f5c792bd" (UID: "a2ba3211-cf44-412e-afad-12b8f5c792bd"). InnerVolumeSpecName "pvc-787026af-147e-408d-99c9-3de7d485501a". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:52:31.155908 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.155887 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ba3211-cf44-412e-afad-12b8f5c792bd-kube-api-access-frm6g" pod "a2ba3211-cf44-412e-afad-12b8f5c792bd" (UID: "a2ba3211-cf44-412e-afad-12b8f5c792bd"). InnerVolumeSpecName "kube-api-access-frm6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:31.226643 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.226583 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6"] Oct 30 18:52:31.226896 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.226849 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" podUID="92e92ebc-c270-46a1-b5fa-11223593671f" containerName="otc-container" containerID="cri-o://497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77" gracePeriod=30 Oct 30 18:52:31.253284 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.253262 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:31.253284 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.253286 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frm6g\" (UniqueName: \"kubernetes.io/projected/a2ba3211-cf44-412e-afad-12b8f5c792bd-kube-api-access-frm6g\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:31.253418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.253296 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-http\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:31.253418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.253321 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:52:31.253418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.253331 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2ba3211-cf44-412e-afad-12b8f5c792bd-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:31.253418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.253340 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-data-science-usage-logs-lokistack-compactor-grpc\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:31.253418 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.253349 5014 reconciler_common.go:299] "Volume detached for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/a2ba3211-cf44-412e-afad-12b8f5c792bd-test-usage-logs-loki-secret\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:31.263374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.263355 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-787026af-147e-408d-99c9-3de7d485501a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:52:31.275825 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.275802 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz"] Oct 30 18:52:31.276174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276158 5014 cpu_manager.go:414] "containerMap: removing container" podUID="09bfdccc-3d3f-49aa-887f-acbe706a0528" containerName="otc-container" Oct 30 18:52:31.276250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276177 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09bfdccc-3d3f-49aa-887f-acbe706a0528" containerName="otc-container" Oct 30 18:52:31.276250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276192 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d0d1316f-fbd8-48d6-b68a-420f873fdc96" containerName="gateway" Oct 30 18:52:31.276250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276200 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d1316f-fbd8-48d6-b68a-420f873fdc96" containerName="gateway" Oct 30 18:52:31.276250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276233 5014 cpu_manager.go:414] "containerMap: removing container" podUID="a2ba3211-cf44-412e-afad-12b8f5c792bd" containerName="loki-compactor" Oct 30 18:52:31.276250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276241 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ba3211-cf44-412e-afad-12b8f5c792bd" containerName="loki-compactor" Oct 30 18:52:31.276524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276258 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d0d1316f-fbd8-48d6-b68a-420f873fdc96" containerName="opa" Oct 30 18:52:31.276524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276265 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d1316f-fbd8-48d6-b68a-420f873fdc96" containerName="opa" Oct 30 18:52:31.276524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276371 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="09bfdccc-3d3f-49aa-887f-acbe706a0528" containerName="otc-container" Oct 30 18:52:31.276524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276384 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d1316f-fbd8-48d6-b68a-420f873fdc96" containerName="gateway" Oct 30 18:52:31.276524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276397 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ba3211-cf44-412e-afad-12b8f5c792bd" containerName="loki-compactor" Oct 30 18:52:31.276524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.276407 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d1316f-fbd8-48d6-b68a-420f873fdc96" containerName="opa" Oct 30 18:52:31.279497 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.279478 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:31.300011 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.299978 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz"] Oct 30 18:52:31.317545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.317521 5014 generic.go:356] "Generic (PLEG): container finished" podID="a2ba3211-cf44-412e-afad-12b8f5c792bd" containerID="e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d" exitCode=0 Oct 30 18:52:31.317663 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.317583 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:31.317663 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.317625 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"a2ba3211-cf44-412e-afad-12b8f5c792bd","Type":"ContainerDied","Data":"e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d"} Oct 30 18:52:31.317792 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.317670 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"a2ba3211-cf44-412e-afad-12b8f5c792bd","Type":"ContainerDied","Data":"b2319bd630e9d2f9cd4503308a4b9d602422f6c0d46ee4d7523c076dbfb021d5"} Oct 30 18:52:31.317792 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.317692 5014 scope.go:122] "RemoveContainer" containerID="e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d" Oct 30 18:52:31.339812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.339794 5014 scope.go:122] "RemoveContainer" containerID="e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d" Oct 30 18:52:31.340115 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:31.340096 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d\": container with ID starting with e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d not found: ID does not exist" containerID="e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d" Oct 30 18:52:31.340181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.340126 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d"} err="failed to get container status \"e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d\": rpc error: code = NotFound desc = could not find container \"e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d\": container with ID starting with e38ecc53213bd249318b17d032bb088c824fe8f390739492ae86dd6f05bde85d not found: ID does not exist" Oct 30 18:52:31.353859 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.353834 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/669e350d-e705-4c97-916e-39a33d52f950-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd859946-hnwdz\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:31.353963 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.353899 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/669e350d-e705-4c97-916e-39a33d52f950-otc-internal\") pod \"data-science-usage-logs-collector-76bd859946-hnwdz\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:31.354092 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.354066 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkzdw\" (UniqueName: \"kubernetes.io/projected/669e350d-e705-4c97-916e-39a33d52f950-kube-api-access-vkzdw\") pod \"data-science-usage-logs-collector-76bd859946-hnwdz\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:31.354149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.354135 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-25f379426d8c7fcf77dc6a7d8d83d8aad1de842552d7dbf772ddd741691b9210\"" Oct 30 18:52:31.374228 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.374199 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:52:31.379331 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.379304 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:52:31.455378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.455299 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/669e350d-e705-4c97-916e-39a33d52f950-otc-internal\") pod \"data-science-usage-logs-collector-76bd859946-hnwdz\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:31.455546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.455393 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkzdw\" (UniqueName: \"kubernetes.io/projected/669e350d-e705-4c97-916e-39a33d52f950-kube-api-access-vkzdw\") pod \"data-science-usage-logs-collector-76bd859946-hnwdz\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:31.455546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.455478 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/669e350d-e705-4c97-916e-39a33d52f950-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd859946-hnwdz\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:31.456003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.455983 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/669e350d-e705-4c97-916e-39a33d52f950-otc-internal\") pod \"data-science-usage-logs-collector-76bd859946-hnwdz\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:31.458169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.458120 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/669e350d-e705-4c97-916e-39a33d52f950-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd859946-hnwdz\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:31.470315 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.470295 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:31.478007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.477986 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkzdw\" (UniqueName: \"kubernetes.io/projected/669e350d-e705-4c97-916e-39a33d52f950-kube-api-access-vkzdw\") pod \"data-science-usage-logs-collector-76bd859946-hnwdz\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:31.556780 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.556751 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e92ebc-c270-46a1-b5fa-11223593671f-kube-api-access-s287n\" (UniqueName: \"kubernetes.io/projected/92e92ebc-c270-46a1-b5fa-11223593671f-kube-api-access-s287n\") pod \"92e92ebc-c270-46a1-b5fa-11223593671f\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " Oct 30 18:52:31.556948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.556794 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e92ebc-c270-46a1-b5fa-11223593671f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/92e92ebc-c270-46a1-b5fa-11223593671f-otc-internal\") pod \"92e92ebc-c270-46a1-b5fa-11223593671f\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " Oct 30 18:52:31.556948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.556859 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92e92ebc-c270-46a1-b5fa-11223593671f-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/92e92ebc-c270-46a1-b5fa-11223593671f-otlp-tls-certs\") pod \"92e92ebc-c270-46a1-b5fa-11223593671f\" (UID: \"92e92ebc-c270-46a1-b5fa-11223593671f\") " Oct 30 18:52:31.557231 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.557211 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e92ebc-c270-46a1-b5fa-11223593671f-otc-internal" pod "92e92ebc-c270-46a1-b5fa-11223593671f" (UID: "92e92ebc-c270-46a1-b5fa-11223593671f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:31.558937 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.558917 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e92ebc-c270-46a1-b5fa-11223593671f-kube-api-access-s287n" pod "92e92ebc-c270-46a1-b5fa-11223593671f" (UID: "92e92ebc-c270-46a1-b5fa-11223593671f"). InnerVolumeSpecName "kube-api-access-s287n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:31.559012 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.558999 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e92ebc-c270-46a1-b5fa-11223593671f-otlp-tls-certs" pod "92e92ebc-c270-46a1-b5fa-11223593671f" (UID: "92e92ebc-c270-46a1-b5fa-11223593671f"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:31.594961 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.594934 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 18:52:31.595370 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.595325 5014 cpu_manager.go:414] "containerMap: removing container" podUID="92e92ebc-c270-46a1-b5fa-11223593671f" containerName="otc-container" Oct 30 18:52:31.595437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.595373 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e92ebc-c270-46a1-b5fa-11223593671f" containerName="otc-container" Oct 30 18:52:31.595494 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.595483 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e92ebc-c270-46a1-b5fa-11223593671f" containerName="otc-container" Oct 30 18:52:31.598641 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.598621 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.602634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.602613 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:31.613137 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.613113 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 30 18:52:31.613245 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.613214 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-grpc\"" Oct 30 18:52:31.613309 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.613268 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-http\"" Oct 30 18:52:31.622500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.622476 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 18:52:31.658750 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.658722 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/92e92ebc-c270-46a1-b5fa-11223593671f-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:31.658895 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.658753 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s287n\" (UniqueName: \"kubernetes.io/projected/92e92ebc-c270-46a1-b5fa-11223593671f-kube-api-access-s287n\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:31.658895 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.658768 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/92e92ebc-c270-46a1-b5fa-11223593671f-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:31.760089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.759996 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.760089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.760044 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.760089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.760074 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.760376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.760202 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.760376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.760252 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmncz\" (UniqueName: \"kubernetes.io/projected/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-kube-api-access-tmncz\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.760376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.760282 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.862178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.861506 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmncz\" (UniqueName: \"kubernetes.io/projected/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-kube-api-access-tmncz\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.862178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.861573 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.862178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.861653 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.862178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.861693 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.862178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.861720 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.862178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.861776 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.862628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.862433 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.862628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.862464 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.864724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.864696 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.864926 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.864911 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.864972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.864957 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.884128 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.884104 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmncz\" (UniqueName: \"kubernetes.io/projected/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-kube-api-access-tmncz\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:31.904442 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.904416 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz"] Oct 30 18:52:31.908554 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:52:31.908527 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669e350d_e705_4c97_916e_39a33d52f950.slice/crio-7fc31f67e3335a0a0ad10f59011cf7d5e95d72de63f5d139d031b9778c8f9132 WatchSource:0}: Error finding container 7fc31f67e3335a0a0ad10f59011cf7d5e95d72de63f5d139d031b9778c8f9132: Status 404 returned error can't find the container with id 7fc31f67e3335a0a0ad10f59011cf7d5e95d72de63f5d139d031b9778c8f9132 Oct 30 18:52:31.910707 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.910680 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:31.910792 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:31.910734 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:32.324888 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.324855 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" event={"ID":"669e350d-e705-4c97-916e-39a33d52f950","Type":"ContainerStarted","Data":"0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50"} Oct 30 18:52:32.325347 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.324903 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" event={"ID":"669e350d-e705-4c97-916e-39a33d52f950","Type":"ContainerStarted","Data":"7fc31f67e3335a0a0ad10f59011cf7d5e95d72de63f5d139d031b9778c8f9132"} Oct 30 18:52:32.326212 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.326185 5014 generic.go:356] "Generic (PLEG): container finished" podID="92e92ebc-c270-46a1-b5fa-11223593671f" containerID="497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77" exitCode=0 Oct 30 18:52:32.326330 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.326220 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" Oct 30 18:52:32.326330 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.326273 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" event={"ID":"92e92ebc-c270-46a1-b5fa-11223593671f","Type":"ContainerDied","Data":"497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77"} Oct 30 18:52:32.326330 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.326317 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6" event={"ID":"92e92ebc-c270-46a1-b5fa-11223593671f","Type":"ContainerDied","Data":"31e8f0f17c86574d624df02a112e97422c2da539948e043208617d2b2bc8f64c"} Oct 30 18:52:32.326441 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.326339 5014 scope.go:122] "RemoveContainer" containerID="497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77" Oct 30 18:52:32.335273 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.335257 5014 scope.go:122] "RemoveContainer" containerID="497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77" Oct 30 18:52:32.335532 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:32.335516 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77\": container with ID starting with 497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77 not found: ID does not exist" containerID="497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77" Oct 30 18:52:32.335629 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.335544 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77"} err="failed to get container status \"497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77\": rpc error: code = NotFound desc = could not find container \"497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77\": container with ID starting with 497f63526eb424adb71cfa7ffd80e7ab844cd4036577a652e71ff9a850b18f77 not found: ID does not exist" Oct 30 18:52:32.365358 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.365314 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" podStartSLOduration=1.365302844 podStartE2EDuration="1.365302844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:32.365253974 +0000 UTC m=+3365.353594460" watchObservedRunningTime="2026-10-30 18:52:32.365302844 +0000 UTC m=+3365.353643313" Oct 30 18:52:32.403919 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.403892 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6"] Oct 30 18:52:32.415096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.415063 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6b8fc67d75-76qc6"] Oct 30 18:52:32.785356 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.785280 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 18:52:32.798438 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:32.798402 5014 csi_plugin.go:220] "Failed to get volume attachment" err="volumeattachments.storage.k8s.io \"csi-7414cac91b824b424fe894433d626f257c573c32523e49fe1ee6ffe352546eec\" is forbidden: User \"system:node:ip-10-0-133-62.ec2.internal\" cannot get resource \"volumeattachments\" in API group \"storage.k8s.io\" at the cluster scope: no relationship found between node 'ip-10-0-133-62.ec2.internal' and this object" attachmentName="csi-7414cac91b824b424fe894433d626f257c573c32523e49fe1ee6ffe352546eec" Oct 30 18:52:32.800334 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.800300 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4"] Oct 30 18:52:32.800705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.800489 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" podUID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerName="gateway" containerID="cri-o://75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5" gracePeriod=30 Oct 30 18:52:32.800705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.800639 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" podUID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerName="opa" containerID="cri-o://6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a" gracePeriod=30 Oct 30 18:52:32.811366 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:32.811334 5014 csi_plugin.go:220] "Failed to get volume attachment" err="volumeattachments.storage.k8s.io \"csi-2d8959d2a75b9a714945a32be7289b244f56b71780476fc260ae0eeed79f1d91\" is forbidden: User \"system:node:ip-10-0-133-62.ec2.internal\" cannot get resource \"volumeattachments\" in API group \"storage.k8s.io\" at the cluster scope: no relationship found between node 'ip-10-0-133-62.ec2.internal' and this object" attachmentName="csi-2d8959d2a75b9a714945a32be7289b244f56b71780476fc260ae0eeed79f1d91" Oct 30 18:52:32.811475 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:32.811391 5014 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[storage wal], unattached volumes=[storage wal], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac" Oct 30 18:52:32.815352 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.815330 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr"] Oct 30 18:52:32.816361 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.815524 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" podUID="f1a87f79-8c4e-416a-9f43-f00e8a29c34f" containerName="loki-distributor" containerID="cri-o://599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51" gracePeriod=30 Oct 30 18:52:32.906453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:32.906423 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz"] Oct 30 18:52:33.141999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.141970 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e92ebc-c270-46a1-b5fa-11223593671f" path="/var/lib/kubelet/pods/92e92ebc-c270-46a1-b5fa-11223593671f/volumes" Oct 30 18:52:33.142377 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.142365 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ba3211-cf44-412e-afad-12b8f5c792bd" path="/var/lib/kubelet/pods/a2ba3211-cf44-412e-afad-12b8f5c792bd/volumes" Oct 30 18:52:33.332088 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.332050 5014 generic.go:356] "Generic (PLEG): container finished" podID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerID="6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a" exitCode=0 Oct 30 18:52:33.332514 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.332120 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" event={"ID":"993c6387-cd33-4c37-b5e9-1e3ac65852c7","Type":"ContainerDied","Data":"6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a"} Oct 30 18:52:33.333203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.333186 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:33.333367 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.333351 5014 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" secret="" err="secret \"data-science-usage-logs-collector-dockercfg-rl5v4\" not found" Oct 30 18:52:33.340476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.340457 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:33.374436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.374415 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-http\") pod \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " Oct 30 18:52:33.374541 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.374450 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-test-loki-lifecycle-secret\") pod \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " Oct 30 18:52:33.374541 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.374481 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-grpc\") pod \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " Oct 30 18:52:33.374939 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:33.374765 5014 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-9d8884b6: configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 30 18:52:33.374939 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:33.374783 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found Oct 30 18:52:33.374939 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:33.374861 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/669e350d-e705-4c97-916e-39a33d52f950-otc-internal podName:669e350d-e705-4c97-916e-39a33d52f950 nodeName:}" failed. No retries permitted until 2026-10-30 18:52:33.874839757 +0000 UTC m=+3366.863180236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/669e350d-e705-4c97-916e-39a33d52f950-otc-internal") pod "data-science-usage-logs-collector-76bd859946-hnwdz" (UID: "669e350d-e705-4c97-916e-39a33d52f950") : configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 30 18:52:33.375147 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:33.375075 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669e350d-e705-4c97-916e-39a33d52f950-otlp-tls-certs podName:669e350d-e705-4c97-916e-39a33d52f950 nodeName:}" failed. No retries permitted until 2026-10-30 18:52:33.875055908 +0000 UTC m=+3366.863396373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/669e350d-e705-4c97-916e-39a33d52f950-otlp-tls-certs") pod "data-science-usage-logs-collector-76bd859946-hnwdz" (UID: "669e350d-e705-4c97-916e-39a33d52f950") : secret "data-science-usage-logs-collector-headless-tls" not found Oct 30 18:52:33.377363 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.377340 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-grpc" pod "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac" (UID: "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:33.377452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.377414 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-http" pod "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac" (UID: "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:33.377452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.377428 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-test-loki-lifecycle-secret" pod "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac" (UID: "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:33.401906 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.401841 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" podUID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerName="opa" probeResult="failure" output="Get \"https://10.132.0.124:8083/ready\": dial tcp 10.132.0.124:8083: connect: connection refused" Oct 30 18:52:33.405675 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.405653 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" podUID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerName="gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:52:33.475611 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.475568 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-kube-api-access-tmncz\" (UniqueName: \"kubernetes.io/projected/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-kube-api-access-tmncz\") pod \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " Oct 30 18:52:33.475722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.475633 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ca-bundle\") pod \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " Oct 30 18:52:33.475722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.475667 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-config\" (UniqueName: \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-config\") pod \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\" (UID: \"7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac\") " Oct 30 18:52:33.475869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.475854 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-grpc\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.475932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.475870 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ingester-http\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.475932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.475880 5014 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-test-loki-lifecycle-secret\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.476055 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.476029 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ca-bundle" pod "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac" (UID: "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:33.476117 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.476040 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-config" pod "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac" (UID: "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:33.477633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.477611 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-kube-api-access-tmncz" pod "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac" (UID: "7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac"). InnerVolumeSpecName "kube-api-access-tmncz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:33.576257 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.576227 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmncz\" (UniqueName: \"kubernetes.io/projected/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-kube-api-access-tmncz\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.576257 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.576255 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.576411 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.576272 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.878697 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:33.878668 5014 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-9d8884b6: configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 30 18:52:33.878846 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:33.878731 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found Oct 30 18:52:33.878846 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:33.878747 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/669e350d-e705-4c97-916e-39a33d52f950-otc-internal podName:669e350d-e705-4c97-916e-39a33d52f950 nodeName:}" failed. No retries permitted until 2026-10-30 18:52:34.878726842 +0000 UTC m=+3367.867067307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/669e350d-e705-4c97-916e-39a33d52f950-otc-internal") pod "data-science-usage-logs-collector-76bd859946-hnwdz" (UID: "669e350d-e705-4c97-916e-39a33d52f950") : configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 30 18:52:33.878846 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:33.878774 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669e350d-e705-4c97-916e-39a33d52f950-otlp-tls-certs podName:669e350d-e705-4c97-916e-39a33d52f950 nodeName:}" failed. No retries permitted until 2026-10-30 18:52:34.878764174 +0000 UTC m=+3367.867104651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/669e350d-e705-4c97-916e-39a33d52f950-otlp-tls-certs") pod "data-science-usage-logs-collector-76bd859946-hnwdz" (UID: "669e350d-e705-4c97-916e-39a33d52f950") : secret "data-science-usage-logs-collector-headless-tls" not found Oct 30 18:52:33.974367 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.974344 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:52:33.979305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.978975 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-kube-api-access-nd44r\" (UniqueName: \"kubernetes.io/projected/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-kube-api-access-nd44r\") pod \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " Oct 30 18:52:33.979305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.979083 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-ca-bundle\") pod \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " Oct 30 18:52:33.979305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.979141 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-grpc\") pod \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " Oct 30 18:52:33.979589 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.979566 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-ca-bundle" pod "f1a87f79-8c4e-416a-9f43-f00e8a29c34f" (UID: "f1a87f79-8c4e-416a-9f43-f00e8a29c34f"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:33.983520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.983497 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-grpc" pod "f1a87f79-8c4e-416a-9f43-f00e8a29c34f" (UID: "f1a87f79-8c4e-416a-9f43-f00e8a29c34f"). InnerVolumeSpecName "data-science-usage-logs-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:33.983648 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:33.983529 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-kube-api-access-nd44r" pod "f1a87f79-8c4e-416a-9f43-f00e8a29c34f" (UID: "f1a87f79-8c4e-416a-9f43-f00e8a29c34f"). InnerVolumeSpecName "kube-api-access-nd44r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:34.042848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.042828 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:52:34.080136 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.080114 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-client-http\") pod \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " Oct 30 18:52:34.080223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.080156 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-rbac\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-rbac\") pod \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " Oct 30 18:52:34.080223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.080177 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tenants\" (UniqueName: \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tenants\") pod \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " Oct 30 18:52:34.080223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.080206 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-lokistack-gateway\") pod \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " Oct 30 18:52:34.080362 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.080233 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-ca-bundle\") pod \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " Oct 30 18:52:34.080362 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.080267 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tls-secret\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tls-secret\") pod \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " Oct 30 18:52:34.080362 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.080304 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-kube-api-access-j48r2\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-kube-api-access-j48r2\") pod \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " Oct 30 18:52:34.080845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.080664 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-ca-bundle" pod "993c6387-cd33-4c37-b5e9-1e3ac65852c7" (UID: "993c6387-cd33-4c37-b5e9-1e3ac65852c7"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:34.081111 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.080772 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-http\") pod \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " Oct 30 18:52:34.081111 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.080990 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-config\" (UniqueName: \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-config\") pod \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\" (UID: \"f1a87f79-8c4e-416a-9f43-f00e8a29c34f\") " Oct 30 18:52:34.081332 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.081311 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.081410 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.081340 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.081410 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.081360 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-grpc\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.081410 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.081375 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd44r\" (UniqueName: \"kubernetes.io/projected/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-kube-api-access-nd44r\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.081824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.081796 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-config" pod "f1a87f79-8c4e-416a-9f43-f00e8a29c34f" (UID: "f1a87f79-8c4e-416a-9f43-f00e8a29c34f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:34.082958 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.082929 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-client-http" pod "993c6387-cd33-4c37-b5e9-1e3ac65852c7" (UID: "993c6387-cd33-4c37-b5e9-1e3ac65852c7"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:34.083111 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.083089 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-kube-api-access-j48r2" pod "993c6387-cd33-4c37-b5e9-1e3ac65852c7" (UID: "993c6387-cd33-4c37-b5e9-1e3ac65852c7"). InnerVolumeSpecName "kube-api-access-j48r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:34.083504 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.083486 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tls-secret" pod "993c6387-cd33-4c37-b5e9-1e3ac65852c7" (UID: "993c6387-cd33-4c37-b5e9-1e3ac65852c7"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:34.083664 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.083587 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-http" pod "f1a87f79-8c4e-416a-9f43-f00e8a29c34f" (UID: "f1a87f79-8c4e-416a-9f43-f00e8a29c34f"). InnerVolumeSpecName "data-science-usage-logs-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:34.085663 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.085640 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-rbac" pod "993c6387-cd33-4c37-b5e9-1e3ac65852c7" (UID: "993c6387-cd33-4c37-b5e9-1e3ac65852c7"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:34.085741 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.085718 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-lokistack-gateway" pod "993c6387-cd33-4c37-b5e9-1e3ac65852c7" (UID: "993c6387-cd33-4c37-b5e9-1e3ac65852c7"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:34.087305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.087279 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tenants" pod "993c6387-cd33-4c37-b5e9-1e3ac65852c7" (UID: "993c6387-cd33-4c37-b5e9-1e3ac65852c7"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:34.182227 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.182200 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\" (UID: \"993c6387-cd33-4c37-b5e9-1e3ac65852c7\") " Oct 30 18:52:34.182423 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.182409 5014 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tls-secret\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.182423 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.182423 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j48r2\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-kube-api-access-j48r2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.182552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.182434 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-data-science-usage-logs-lokistack-distributor-http\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.182552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.182445 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f1a87f79-8c4e-416a-9f43-f00e8a29c34f-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.182552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.182454 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-client-http\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.182552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.182462 5014 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.182552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.182471 5014 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/993c6387-cd33-4c37-b5e9-1e3ac65852c7-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.182552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.182478 5014 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/993c6387-cd33-4c37-b5e9-1e3ac65852c7-lokistack-gateway\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.184163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.184139 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-ca-bundle" pod "993c6387-cd33-4c37-b5e9-1e3ac65852c7" (UID: "993c6387-cd33-4c37-b5e9-1e3ac65852c7"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:34.283647 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.283621 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/993c6387-cd33-4c37-b5e9-1e3ac65852c7-data-science-usage-logs-lokistack-gateway-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.339337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.339307 5014 generic.go:356] "Generic (PLEG): container finished" podID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerID="75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5" exitCode=0 Oct 30 18:52:34.339742 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.339352 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" Oct 30 18:52:34.339742 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.339400 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" event={"ID":"993c6387-cd33-4c37-b5e9-1e3ac65852c7","Type":"ContainerDied","Data":"75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5"} Oct 30 18:52:34.339742 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.339443 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4" event={"ID":"993c6387-cd33-4c37-b5e9-1e3ac65852c7","Type":"ContainerDied","Data":"5ddc8e9092ef45f916bd1f5907cf8d245ece39606f2eb1f2e19b62d561531c5d"} Oct 30 18:52:34.339742 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.339462 5014 scope.go:122] "RemoveContainer" containerID="6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a" Oct 30 18:52:34.341054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.341024 5014 generic.go:356] "Generic (PLEG): container finished" podID="f1a87f79-8c4e-416a-9f43-f00e8a29c34f" containerID="599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51" exitCode=0 Oct 30 18:52:34.341165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.341064 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" Oct 30 18:52:34.341165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.341107 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" event={"ID":"f1a87f79-8c4e-416a-9f43-f00e8a29c34f","Type":"ContainerDied","Data":"599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51"} Oct 30 18:52:34.341165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.341139 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr" event={"ID":"f1a87f79-8c4e-416a-9f43-f00e8a29c34f","Type":"ContainerDied","Data":"650d5e9cfd752ae82f1fb1240f2545b5fd128622c3d42a6c12f711987a3a03db"} Oct 30 18:52:34.341344 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.341326 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:34.341409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.341396 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" podUID="669e350d-e705-4c97-916e-39a33d52f950" containerName="otc-container" containerID="cri-o://0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50" gracePeriod=30 Oct 30 18:52:34.357457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.357436 5014 scope.go:122] "RemoveContainer" containerID="75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5" Oct 30 18:52:34.453228 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.453206 5014 scope.go:122] "RemoveContainer" containerID="6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a" Oct 30 18:52:34.453525 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:34.453495 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a\": container with ID starting with 6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a not found: ID does not exist" containerID="6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a" Oct 30 18:52:34.453587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.453525 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a"} err="failed to get container status \"6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a\": rpc error: code = NotFound desc = could not find container \"6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a\": container with ID starting with 6c319ac924552c20a9a2994b53344858d95c0a9d2ff22859ba57c2332e21d15a not found: ID does not exist" Oct 30 18:52:34.453587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.453542 5014 scope.go:122] "RemoveContainer" containerID="75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5" Oct 30 18:52:34.453812 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:34.453793 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5\": container with ID starting with 75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5 not found: ID does not exist" containerID="75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5" Oct 30 18:52:34.453852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.453818 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5"} err="failed to get container status \"75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5\": rpc error: code = NotFound desc = could not find container \"75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5\": container with ID starting with 75d0d60f59b8d881c7217b79f377c9d45889047cda72b2db9c1ea86f923435f5 not found: ID does not exist" Oct 30 18:52:34.453852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.453833 5014 scope.go:122] "RemoveContainer" containerID="599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51" Oct 30 18:52:34.461488 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.461422 5014 scope.go:122] "RemoveContainer" containerID="599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51" Oct 30 18:52:34.461699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.461681 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4"] Oct 30 18:52:34.461763 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:34.461685 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51\": container with ID starting with 599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51 not found: ID does not exist" containerID="599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51" Oct 30 18:52:34.461763 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.461736 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51"} err="failed to get container status \"599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51\": rpc error: code = NotFound desc = could not find container \"599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51\": container with ID starting with 599731bcf16b093ec0baf846277ade455daaded44b0b3490a676024f4d331d51 not found: ID does not exist" Oct 30 18:52:34.473174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.473151 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-l7cn4"] Oct 30 18:52:34.499155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.499132 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr"] Oct 30 18:52:34.505436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.505411 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j9zvr"] Oct 30 18:52:34.567639 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.567614 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 18:52:34.571973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.571951 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 18:52:34.591492 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.591448 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:34.687218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.687194 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669e350d-e705-4c97-916e-39a33d52f950-kube-api-access-vkzdw\" (UniqueName: \"kubernetes.io/projected/669e350d-e705-4c97-916e-39a33d52f950-kube-api-access-vkzdw\") pod \"669e350d-e705-4c97-916e-39a33d52f950\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " Oct 30 18:52:34.687360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.687252 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/669e350d-e705-4c97-916e-39a33d52f950-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/669e350d-e705-4c97-916e-39a33d52f950-otlp-tls-certs\") pod \"669e350d-e705-4c97-916e-39a33d52f950\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " Oct 30 18:52:34.687360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.687286 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669e350d-e705-4c97-916e-39a33d52f950-otc-internal\" (UniqueName: \"kubernetes.io/configmap/669e350d-e705-4c97-916e-39a33d52f950-otc-internal\") pod \"669e350d-e705-4c97-916e-39a33d52f950\" (UID: \"669e350d-e705-4c97-916e-39a33d52f950\") " Oct 30 18:52:34.687743 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.687714 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669e350d-e705-4c97-916e-39a33d52f950-otc-internal" pod "669e350d-e705-4c97-916e-39a33d52f950" (UID: "669e350d-e705-4c97-916e-39a33d52f950"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:34.689529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.689505 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669e350d-e705-4c97-916e-39a33d52f950-kube-api-access-vkzdw" pod "669e350d-e705-4c97-916e-39a33d52f950" (UID: "669e350d-e705-4c97-916e-39a33d52f950"). InnerVolumeSpecName "kube-api-access-vkzdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:34.689658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.689545 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669e350d-e705-4c97-916e-39a33d52f950-otlp-tls-certs" pod "669e350d-e705-4c97-916e-39a33d52f950" (UID: "669e350d-e705-4c97-916e-39a33d52f950"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:34.787869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.787840 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkzdw\" (UniqueName: \"kubernetes.io/projected/669e350d-e705-4c97-916e-39a33d52f950-kube-api-access-vkzdw\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.787869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.787865 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/669e350d-e705-4c97-916e-39a33d52f950-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.787992 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:34.787877 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/669e350d-e705-4c97-916e-39a33d52f950-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:52:35.141764 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.141737 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac" path="/var/lib/kubelet/pods/7e89e472-3dd3-439c-b8a6-e4a3fb6a18ac/volumes" Oct 30 18:52:35.142020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.142008 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" path="/var/lib/kubelet/pods/993c6387-cd33-4c37-b5e9-1e3ac65852c7/volumes" Oct 30 18:52:35.142399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.142386 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a87f79-8c4e-416a-9f43-f00e8a29c34f" path="/var/lib/kubelet/pods/f1a87f79-8c4e-416a-9f43-f00e8a29c34f/volumes" Oct 30 18:52:35.347257 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.347225 5014 generic.go:356] "Generic (PLEG): container finished" podID="669e350d-e705-4c97-916e-39a33d52f950" containerID="0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50" exitCode=0 Oct 30 18:52:35.347647 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.347277 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" Oct 30 18:52:35.347647 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.347314 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" event={"ID":"669e350d-e705-4c97-916e-39a33d52f950","Type":"ContainerDied","Data":"0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50"} Oct 30 18:52:35.347647 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.347368 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz" event={"ID":"669e350d-e705-4c97-916e-39a33d52f950","Type":"ContainerDied","Data":"7fc31f67e3335a0a0ad10f59011cf7d5e95d72de63f5d139d031b9778c8f9132"} Oct 30 18:52:35.347647 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.347401 5014 scope.go:122] "RemoveContainer" containerID="0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50" Oct 30 18:52:35.356137 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.356120 5014 scope.go:122] "RemoveContainer" containerID="0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50" Oct 30 18:52:35.356427 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:35.356402 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50\": container with ID starting with 0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50 not found: ID does not exist" containerID="0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50" Oct 30 18:52:35.356499 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.356441 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50"} err="failed to get container status \"0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50\": rpc error: code = NotFound desc = could not find container \"0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50\": container with ID starting with 0e016c4097b9384153bb00dacb49a980804f8c00c490abbf0c11e73d94fd6e50 not found: ID does not exist" Oct 30 18:52:35.396999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.396945 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz"] Oct 30 18:52:35.408642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:35.408621 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-hnwdz"] Oct 30 18:52:37.142640 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:37.142581 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669e350d-e705-4c97-916e-39a33d52f950" path="/var/lib/kubelet/pods/669e350d-e705-4c97-916e-39a33d52f950/volumes" Oct 30 18:52:38.308307 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.308272 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4"] Oct 30 18:52:38.308911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.308838 5014 cpu_manager.go:414] "containerMap: removing container" podUID="f1a87f79-8c4e-416a-9f43-f00e8a29c34f" containerName="loki-distributor" Oct 30 18:52:38.308911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.308860 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a87f79-8c4e-416a-9f43-f00e8a29c34f" containerName="loki-distributor" Oct 30 18:52:38.308911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.308872 5014 cpu_manager.go:414] "containerMap: removing container" podUID="669e350d-e705-4c97-916e-39a33d52f950" containerName="otc-container" Oct 30 18:52:38.308911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.308877 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669e350d-e705-4c97-916e-39a33d52f950" containerName="otc-container" Oct 30 18:52:38.308911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.308893 5014 cpu_manager.go:414] "containerMap: removing container" podUID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerName="opa" Oct 30 18:52:38.308911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.308902 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerName="opa" Oct 30 18:52:38.309156 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.308920 5014 cpu_manager.go:414] "containerMap: removing container" podUID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerName="gateway" Oct 30 18:52:38.309156 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.308928 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerName="gateway" Oct 30 18:52:38.309156 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.309012 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a87f79-8c4e-416a-9f43-f00e8a29c34f" containerName="loki-distributor" Oct 30 18:52:38.309156 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.309023 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="669e350d-e705-4c97-916e-39a33d52f950" containerName="otc-container" Oct 30 18:52:38.309156 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.309032 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerName="opa" Oct 30 18:52:38.309156 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.309038 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="993c6387-cd33-4c37-b5e9-1e3ac65852c7" containerName="gateway" Oct 30 18:52:38.313825 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.313800 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.329332 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.329310 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-distributor-grpc\"" Oct 30 18:52:38.332180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.332160 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-distributor-http\"" Oct 30 18:52:38.340989 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.340970 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ca-bundle\"" Oct 30 18:52:38.341094 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.340987 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:52:38.343753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.343726 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-config\"" Oct 30 18:52:38.343835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.343816 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:52:38.345407 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.345388 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4"] Oct 30 18:52:38.398580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.398554 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 18:52:38.401948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.401929 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.410622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.410579 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.410740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.410674 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.410740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.410722 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.410884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.410857 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.410955 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.410918 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdj6v\" (UniqueName: \"kubernetes.io/projected/0efed69a-48e3-4558-8e39-15798caf4908-kube-api-access-vdj6v\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.425056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.425033 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 18:52:38.429658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.429635 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 30 18:52:38.430800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.430778 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-grpc\"" Oct 30 18:52:38.430913 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.430897 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-dockercfg-hb59n\"" Oct 30 18:52:38.431004 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.430984 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-http\"" Oct 30 18:52:38.508563 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.508534 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj"] Oct 30 18:52:38.517045 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.517010 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.517255 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.517236 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htkb7\" (UniqueName: \"kubernetes.io/projected/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-kube-api-access-htkb7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.517376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.517361 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.517508 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.517495 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.517657 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.517628 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.517792 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.517778 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdj6v\" (UniqueName: \"kubernetes.io/projected/0efed69a-48e3-4558-8e39-15798caf4908-kube-api-access-vdj6v\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.517909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.517890 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.518036 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.518023 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.518195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.518181 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.518304 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.518289 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.518410 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.518395 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.519537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.519515 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.520548 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.520525 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.522824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.522800 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.523449 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.523419 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.523759 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.523735 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.537809 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.537754 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj"] Oct 30 18:52:38.538722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.538702 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-http\"" Oct 30 18:52:38.538845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.538828 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-grpc\"" Oct 30 18:52:38.544119 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.544095 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdj6v\" (UniqueName: \"kubernetes.io/projected/0efed69a-48e3-4558-8e39-15798caf4908-kube-api-access-vdj6v\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.619678 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.619609 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.619678 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.619655 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxwx9\" (UniqueName: \"kubernetes.io/projected/8c4f2111-4745-41bf-a1f7-a1fe6217260c-kube-api-access-cxwx9\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.619816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.619684 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.619816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.619715 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.619816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.619746 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.619816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.619792 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.620031 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.619824 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.620031 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.619965 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.620031 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.620012 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.620185 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.620075 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.620185 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.620114 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.620185 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.620143 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htkb7\" (UniqueName: \"kubernetes.io/projected/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-kube-api-access-htkb7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.620557 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.620532 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.620779 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.620753 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.622780 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.622759 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.622943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.622920 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.623176 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.623160 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.642575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.642552 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:38.644728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.644711 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htkb7\" (UniqueName: \"kubernetes.io/projected/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-kube-api-access-htkb7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:38.721080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.721054 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.721211 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.721118 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.721211 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.721172 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.721211 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.721194 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.721316 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.721230 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxwx9\" (UniqueName: \"kubernetes.io/projected/8c4f2111-4745-41bf-a1f7-a1fe6217260c-kube-api-access-cxwx9\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.721316 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.721269 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.721945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.721925 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.722163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.722144 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.725019 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.724975 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.725282 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.725241 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.726020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.726002 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.745782 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.745761 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxwx9\" (UniqueName: \"kubernetes.io/projected/8c4f2111-4745-41bf-a1f7-a1fe6217260c-kube-api-access-cxwx9\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.860601 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.860572 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:38.935373 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.935345 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr"] Oct 30 18:52:38.942985 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.942961 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:38.945329 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.944704 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4"] Oct 30 18:52:38.959953 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:52:38.959922 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0efed69a_48e3_4558_8e39_15798caf4908.slice/crio-a6e1e4fe99fdec1717b1010672efff0f2fc7774b0e4d700e07648d22b9a660d4 WatchSource:0}: Error finding container a6e1e4fe99fdec1717b1010672efff0f2fc7774b0e4d700e07648d22b9a660d4: Status 404 returned error can't find the container with id a6e1e4fe99fdec1717b1010672efff0f2fc7774b0e4d700e07648d22b9a660d4 Oct 30 18:52:38.963121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.963070 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 18:52:38.963248 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.963213 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 18:52:38.963292 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.963266 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-ca-bundle\"" Oct 30 18:52:38.963327 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.963309 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-http\"" Oct 30 18:52:38.963874 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.963740 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-client-http\"" Oct 30 18:52:38.969159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:38.968824 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr"] Oct 30 18:52:39.023795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.023773 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.023926 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.023812 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-tenants\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.023926 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.023834 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.023926 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.023889 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq9kc\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-kube-api-access-kq9kc\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.024054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.024000 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.024054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.024035 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-rbac\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.024211 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.024194 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.024278 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.024243 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.125295 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.125219 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-tenants\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.125295 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.125257 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.125295 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.125279 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9kc\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-kube-api-access-kq9kc\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.125581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.125323 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.125581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.125350 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-rbac\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.125581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.125394 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.125581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.125446 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.125581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.125477 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.126251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.126222 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.126453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.126428 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-rbac\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.126911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.126885 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.127984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.127964 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-tenants\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.128078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.127983 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.128343 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.128318 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.128396 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.128365 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.148771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.148750 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9kc\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-kube-api-access-kq9kc\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.160450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.160415 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj"] Oct 30 18:52:39.164917 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:52:39.164895 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c4f2111_4745_41bf_a1f7_a1fe6217260c.slice/crio-0ec2e9c4339438f29ebdde26279a5b0ff6e045ac264f80eaeb8fb68fa9764dd9 WatchSource:0}: Error finding container 0ec2e9c4339438f29ebdde26279a5b0ff6e045ac264f80eaeb8fb68fa9764dd9: Status 404 returned error can't find the container with id 0ec2e9c4339438f29ebdde26279a5b0ff6e045ac264f80eaeb8fb68fa9764dd9 Oct 30 18:52:39.277661 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.277625 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:39.371446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.371407 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" event={"ID":"8c4f2111-4745-41bf-a1f7-a1fe6217260c","Type":"ContainerStarted","Data":"8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065"} Oct 30 18:52:39.371949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.371458 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" event={"ID":"8c4f2111-4745-41bf-a1f7-a1fe6217260c","Type":"ContainerStarted","Data":"0ec2e9c4339438f29ebdde26279a5b0ff6e045ac264f80eaeb8fb68fa9764dd9"} Oct 30 18:52:39.371949 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.371526 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:52:39.373031 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.373007 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" event={"ID":"0efed69a-48e3-4558-8e39-15798caf4908","Type":"ContainerStarted","Data":"71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c"} Oct 30 18:52:39.373165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.373044 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:52:39.373165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.373057 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" event={"ID":"0efed69a-48e3-4558-8e39-15798caf4908","Type":"ContainerStarted","Data":"a6e1e4fe99fdec1717b1010672efff0f2fc7774b0e4d700e07648d22b9a660d4"} Oct 30 18:52:39.413583 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.413498 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" podStartSLOduration=1.413484927 podStartE2EDuration="1.413484927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:39.412516754 +0000 UTC m=+3372.400857242" watchObservedRunningTime="2026-10-30 18:52:39.413484927 +0000 UTC m=+3372.401825413" Oct 30 18:52:39.457797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.457754 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" podStartSLOduration=1.457743752 podStartE2EDuration="1.457743752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:39.456438067 +0000 UTC m=+3372.444778535" watchObservedRunningTime="2026-10-30 18:52:39.457743752 +0000 UTC m=+3372.446084239" Oct 30 18:52:39.591178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:39.591150 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr"] Oct 30 18:52:39.596082 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:52:39.596048 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe024e78_834e_450a_99b6_d40ac786e0d4.slice/crio-bf298176720dc7210ba939b63caf897fc9b3f9ded8e14ce9167cd1ffc9228eb8 WatchSource:0}: Error finding container bf298176720dc7210ba939b63caf897fc9b3f9ded8e14ce9167cd1ffc9228eb8: Status 404 returned error can't find the container with id bf298176720dc7210ba939b63caf897fc9b3f9ded8e14ce9167cd1ffc9228eb8 Oct 30 18:52:40.379281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:40.379241 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" event={"ID":"fe024e78-834e-450a-99b6-d40ac786e0d4","Type":"ContainerStarted","Data":"edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc"} Oct 30 18:52:40.379769 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:40.379749 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:40.379869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:40.379785 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:40.379869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:40.379801 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" event={"ID":"fe024e78-834e-450a-99b6-d40ac786e0d4","Type":"ContainerStarted","Data":"f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1"} Oct 30 18:52:40.379869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:40.379819 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" event={"ID":"fe024e78-834e-450a-99b6-d40ac786e0d4","Type":"ContainerStarted","Data":"bf298176720dc7210ba939b63caf897fc9b3f9ded8e14ce9167cd1ffc9228eb8"} Oct 30 18:52:40.385744 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:40.385724 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:40.388310 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:40.388290 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:52:40.424285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:40.424239 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" podStartSLOduration=2.424226005 podStartE2EDuration="2.424226005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:40.422526135 +0000 UTC m=+3373.410866622" watchObservedRunningTime="2026-10-30 18:52:40.424226005 +0000 UTC m=+3373.412566493" Oct 30 18:52:43.334621 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:52:43.334564 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:52:48.907382 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:48.907348 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9bd50dcf-f652-43c4-bf16-dd7f49cb7f7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:48.935734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:48.935706 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-9bd50dcf-f652-43c4-bf16-dd7f49cb7f7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:49.008202 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:49.008178 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9bd50dcf-f652-43c4-bf16-dd7f49cb7f7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:49.008320 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:49.008274 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9bd50dcf-f652-43c4-bf16-dd7f49cb7f7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:49.022363 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:49.022341 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9bd50dcf-f652-43c4-bf16-dd7f49cb7f7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") DevicePath \"csi-7414cac91b824b424fe894433d626f257c573c32523e49fe1ee6ffe352546eec\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:49.123085 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:49.123046 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9bd50dcf-f652-43c4-bf16-dd7f49cb7f7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/c59e0333bfacae8243fcdbdeca9ad6e70667a5474ab3176a1416861c75db702d/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:49.155459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:49.155438 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-9bd50dcf-f652-43c4-bf16-dd7f49cb7f7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:51.935075 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:51.935032 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f16319de-02a0-44df-9fbf-4a175d4ad3b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:51.963266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:51.963240 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-f16319de-02a0-44df-9fbf-4a175d4ad3b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:52.035638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:52.035606 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f16319de-02a0-44df-9fbf-4a175d4ad3b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:52.035743 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:52.035709 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f16319de-02a0-44df-9fbf-4a175d4ad3b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:52.049756 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:52.049731 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f16319de-02a0-44df-9fbf-4a175d4ad3b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") DevicePath \"csi-2d8959d2a75b9a714945a32be7289b244f56b71780476fc260ae0eeed79f1d91\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:52.149507 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:52.149483 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f16319de-02a0-44df-9fbf-4a175d4ad3b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8d67a4309d5c992166a3eb1f76372dee84e9d17e695a02ef5687339f8e481ef0/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:52.175525 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:52.175498 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-f16319de-02a0-44df-9fbf-4a175d4ad3b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:52.216344 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:52.216272 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:52.527335 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:52.527260 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 18:52:52.530731 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:52:52.530703 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod583a0cd2_a73f_4ecf_9949_5a1cf9d659bc.slice/crio-1f219ef3d568d01fd78e62044ef05ae9dcb2caf738f7239893427d2c27d870f5 WatchSource:0}: Error finding container 1f219ef3d568d01fd78e62044ef05ae9dcb2caf738f7239893427d2c27d870f5: Status 404 returned error can't find the container with id 1f219ef3d568d01fd78e62044ef05ae9dcb2caf738f7239893427d2c27d870f5 Oct 30 18:52:53.442534 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:53.442497 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc","Type":"ContainerStarted","Data":"1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f"} Oct 30 18:52:53.443066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:53.442543 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc","Type":"ContainerStarted","Data":"1f219ef3d568d01fd78e62044ef05ae9dcb2caf738f7239893427d2c27d870f5"} Oct 30 18:52:53.443066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:53.442688 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:52:53.492660 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:52:53.492609 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podStartSLOduration=15.492576738 podStartE2EDuration="15.492576738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:53.486127896 +0000 UTC m=+3386.474468383" watchObservedRunningTime="2026-10-30 18:52:53.492576738 +0000 UTC m=+3386.480917225" Oct 30 18:53:00.385579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:00.385546 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:53:00.386015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:00.385995 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:53:13.333731 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:53:13.333689 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:53:14.452170 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:14.452134 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:53:24.452221 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:24.452181 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:53:34.451790 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:34.451757 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:53:43.331839 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:53:43.331799 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:53:46.786943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:46.786910 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 18:53:46.787774 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:46.787743 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" containerName="loki-ingester" containerID="cri-o://1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f" gracePeriod=600 Oct 30 18:53:46.815294 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:46.815258 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj"] Oct 30 18:53:46.815525 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:46.815482 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" podUID="8c4f2111-4745-41bf-a1f7-a1fe6217260c" containerName="loki-querier" containerID="cri-o://8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065" gracePeriod=30 Oct 30 18:53:46.833873 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:46.833845 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4"] Oct 30 18:53:46.834082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:46.834060 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" podUID="0efed69a-48e3-4558-8e39-15798caf4908" containerName="loki-distributor" containerID="cri-o://71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c" gracePeriod=30 Oct 30 18:53:46.846082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:46.846054 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr"] Oct 30 18:53:46.846274 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:46.846237 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" podUID="fe024e78-834e-450a-99b6-d40ac786e0d4" containerName="gateway" containerID="cri-o://f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1" gracePeriod=30 Oct 30 18:53:46.846367 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:46.846254 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" podUID="fe024e78-834e-450a-99b6-d40ac786e0d4" containerName="opa" containerID="cri-o://edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc" gracePeriod=30 Oct 30 18:53:47.371121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.371093 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:53:47.448837 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.448773 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-ca-bundle\") pod \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " Oct 30 18:53:47.448972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.448832 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-grpc\") pod \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " Oct 30 18:53:47.448972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.448879 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-test-loki-lifecycle-secret\") pod \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " Oct 30 18:53:47.448972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.448906 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-config\" (UniqueName: \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-config\") pod \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " Oct 30 18:53:47.448972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.448967 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4f2111-4745-41bf-a1f7-a1fe6217260c-kube-api-access-cxwx9\" (UniqueName: \"kubernetes.io/projected/8c4f2111-4745-41bf-a1f7-a1fe6217260c-kube-api-access-cxwx9\") pod \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " Oct 30 18:53:47.449188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.449019 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-http\") pod \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\" (UID: \"8c4f2111-4745-41bf-a1f7-a1fe6217260c\") " Oct 30 18:53:47.449188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.449118 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-ca-bundle" pod "8c4f2111-4745-41bf-a1f7-a1fe6217260c" (UID: "8c4f2111-4745-41bf-a1f7-a1fe6217260c"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:47.449402 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.449375 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-config" pod "8c4f2111-4745-41bf-a1f7-a1fe6217260c" (UID: "8c4f2111-4745-41bf-a1f7-a1fe6217260c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:47.449486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.449462 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.449550 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.449485 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c4f2111-4745-41bf-a1f7-a1fe6217260c-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.451258 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.451233 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-grpc" pod "8c4f2111-4745-41bf-a1f7-a1fe6217260c" (UID: "8c4f2111-4745-41bf-a1f7-a1fe6217260c"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:47.451353 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.451262 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-test-loki-lifecycle-secret" pod "8c4f2111-4745-41bf-a1f7-a1fe6217260c" (UID: "8c4f2111-4745-41bf-a1f7-a1fe6217260c"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:47.451353 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.451274 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4f2111-4745-41bf-a1f7-a1fe6217260c-kube-api-access-cxwx9" pod "8c4f2111-4745-41bf-a1f7-a1fe6217260c" (UID: "8c4f2111-4745-41bf-a1f7-a1fe6217260c"). InnerVolumeSpecName "kube-api-access-cxwx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:47.451757 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.451735 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-http" pod "8c4f2111-4745-41bf-a1f7-a1fe6217260c" (UID: "8c4f2111-4745-41bf-a1f7-a1fe6217260c"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:47.550130 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.550108 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-grpc\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.550130 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.550130 5014 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-test-loki-lifecycle-secret\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.550262 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.550140 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxwx9\" (UniqueName: \"kubernetes.io/projected/8c4f2111-4745-41bf-a1f7-a1fe6217260c-kube-api-access-cxwx9\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.550262 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.550149 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/8c4f2111-4745-41bf-a1f7-a1fe6217260c-data-science-usage-logs-lokistack-querier-http\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.697374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.697344 5014 generic.go:356] "Generic (PLEG): container finished" podID="fe024e78-834e-450a-99b6-d40ac786e0d4" containerID="edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc" exitCode=0 Oct 30 18:53:47.697500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.697412 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" event={"ID":"fe024e78-834e-450a-99b6-d40ac786e0d4","Type":"ContainerDied","Data":"edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc"} Oct 30 18:53:47.698816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.698725 5014 generic.go:356] "Generic (PLEG): container finished" podID="8c4f2111-4745-41bf-a1f7-a1fe6217260c" containerID="8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065" exitCode=0 Oct 30 18:53:47.698816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.698801 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" event={"ID":"8c4f2111-4745-41bf-a1f7-a1fe6217260c","Type":"ContainerDied","Data":"8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065"} Oct 30 18:53:47.698965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.698838 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" event={"ID":"8c4f2111-4745-41bf-a1f7-a1fe6217260c","Type":"ContainerDied","Data":"0ec2e9c4339438f29ebdde26279a5b0ff6e045ac264f80eaeb8fb68fa9764dd9"} Oct 30 18:53:47.698965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.698861 5014 scope.go:122] "RemoveContainer" containerID="8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065" Oct 30 18:53:47.699061 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.698991 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj" Oct 30 18:53:47.711484 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.710956 5014 scope.go:122] "RemoveContainer" containerID="8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065" Oct 30 18:53:47.711693 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:53:47.711662 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065\": container with ID starting with 8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065 not found: ID does not exist" containerID="8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065" Oct 30 18:53:47.711781 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.711706 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065"} err="failed to get container status \"8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065\": rpc error: code = NotFound desc = could not find container \"8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065\": container with ID starting with 8a55153035acb718e0ab34b63cba721586a92024dddef48f05ae017c298f7065 not found: ID does not exist" Oct 30 18:53:47.753041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.753020 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj"] Oct 30 18:53:47.757425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.757403 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-rxgwj"] Oct 30 18:53:47.930251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.930223 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 18:53:47.934800 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:47.934775 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 18:53:48.110497 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.110467 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:53:48.114035 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.114016 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:53:48.204174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.204111 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:53:48.254975 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.254945 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-http\") pod \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " Oct 30 18:53:48.255100 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.254989 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-http\") pod \"0efed69a-48e3-4558-8e39-15798caf4908\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " Oct 30 18:53:48.255100 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.255073 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7\") pod \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " Oct 30 18:53:48.255100 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.255093 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-config\" (UniqueName: \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-config\") pod \"0efed69a-48e3-4558-8e39-15798caf4908\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " Oct 30 18:53:48.255235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.255108 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-lokistack-gateway\") pod \"fe024e78-834e-450a-99b6-d40ac786e0d4\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " Oct 30 18:53:48.255235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.255139 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-ca-bundle\") pod \"fe024e78-834e-450a-99b6-d40ac786e0d4\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " Oct 30 18:53:48.255235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.255172 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-kube-api-access-htkb7\" (UniqueName: \"kubernetes.io/projected/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-kube-api-access-htkb7\") pod \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " Oct 30 18:53:48.255235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.255195 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-kube-api-access-kq9kc\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-kube-api-access-kq9kc\") pod \"fe024e78-834e-450a-99b6-d40ac786e0d4\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " Oct 30 18:53:48.255452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.255239 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"fe024e78-834e-450a-99b6-d40ac786e0d4\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.255751 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-config" pod "0efed69a-48e3-4558-8e39-15798caf4908" (UID: "0efed69a-48e3-4558-8e39-15798caf4908"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.255837 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-ca-bundle" pod "fe024e78-834e-450a-99b6-d40ac786e0d4" (UID: "fe024e78-834e-450a-99b6-d40ac786e0d4"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256056 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-ca-bundle\") pod \"0efed69a-48e3-4558-8e39-15798caf4908\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256097 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-tenants\" (UniqueName: \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-tenants\") pod \"fe024e78-834e-450a-99b6-d40ac786e0d4\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256144 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-test-loki-lifecycle-secret\") pod \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256230 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac\") pod \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256275 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0efed69a-48e3-4558-8e39-15798caf4908-kube-api-access-vdj6v\" (UniqueName: \"kubernetes.io/projected/0efed69a-48e3-4558-8e39-15798caf4908-kube-api-access-vdj6v\") pod \"0efed69a-48e3-4558-8e39-15798caf4908\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256353 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-grpc\") pod \"0efed69a-48e3-4558-8e39-15798caf4908\" (UID: \"0efed69a-48e3-4558-8e39-15798caf4908\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256387 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-tls-secret\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-tls-secret\") pod \"fe024e78-834e-450a-99b6-d40ac786e0d4\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256431 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-client-http\") pod \"fe024e78-834e-450a-99b6-d40ac786e0d4\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256468 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-config\" (UniqueName: \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-config\") pod \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256498 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ca-bundle\") pod \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256526 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-rbac\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-rbac\") pod \"fe024e78-834e-450a-99b6-d40ac786e0d4\" (UID: \"fe024e78-834e-450a-99b6-d40ac786e0d4\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256563 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-grpc\") pod \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\" (UID: \"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc\") " Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256936 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.256960 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.258115 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-kube-api-access-htkb7" pod "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" (UID: "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc"). InnerVolumeSpecName "kube-api-access-htkb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.258132 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-http" pod "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" (UID: "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.258210 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-http" pod "0efed69a-48e3-4558-8e39-15798caf4908" (UID: "0efed69a-48e3-4558-8e39-15798caf4908"). InnerVolumeSpecName "data-science-usage-logs-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.258571 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-ca-bundle" pod "fe024e78-834e-450a-99b6-d40ac786e0d4" (UID: "fe024e78-834e-450a-99b6-d40ac786e0d4"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.258635 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-config" pod "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" (UID: "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.258860 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-kube-api-access-kq9kc" pod "fe024e78-834e-450a-99b6-d40ac786e0d4" (UID: "fe024e78-834e-450a-99b6-d40ac786e0d4"). InnerVolumeSpecName "kube-api-access-kq9kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.258962 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-ca-bundle" pod "0efed69a-48e3-4558-8e39-15798caf4908" (UID: "0efed69a-48e3-4558-8e39-15798caf4908"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.259348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.259334 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ca-bundle" pod "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" (UID: "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.261556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.260706 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-test-loki-lifecycle-secret" pod "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" (UID: "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:48.261556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.260793 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7" pod "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" (UID: "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc"). InnerVolumeSpecName "pvc-9bd50dcf-f652-43c4-bf16-dd7f49cb7f7e". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:53:48.261556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.261288 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-client-http" pod "fe024e78-834e-450a-99b6-d40ac786e0d4" (UID: "fe024e78-834e-450a-99b6-d40ac786e0d4"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:48.261556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.261435 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-grpc" pod "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" (UID: "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:48.261925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.261899 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-tls-secret" pod "fe024e78-834e-450a-99b6-d40ac786e0d4" (UID: "fe024e78-834e-450a-99b6-d40ac786e0d4"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:48.263348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.263326 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac" pod "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" (UID: "583a0cd2-a73f-4ecf-9949-5a1cf9d659bc"). InnerVolumeSpecName "pvc-f16319de-02a0-44df-9fbf-4a175d4ad3b9". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:53:48.263700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.263675 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-grpc" pod "0efed69a-48e3-4558-8e39-15798caf4908" (UID: "0efed69a-48e3-4558-8e39-15798caf4908"). InnerVolumeSpecName "data-science-usage-logs-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:48.263784 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.263705 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efed69a-48e3-4558-8e39-15798caf4908-kube-api-access-vdj6v" pod "0efed69a-48e3-4558-8e39-15798caf4908" (UID: "0efed69a-48e3-4558-8e39-15798caf4908"). InnerVolumeSpecName "kube-api-access-vdj6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:48.264459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.264438 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-lokistack-gateway" pod "fe024e78-834e-450a-99b6-d40ac786e0d4" (UID: "fe024e78-834e-450a-99b6-d40ac786e0d4"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.267241 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.267221 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-tenants" pod "fe024e78-834e-450a-99b6-d40ac786e0d4" (UID: "fe024e78-834e-450a-99b6-d40ac786e0d4"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:48.267538 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.267519 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-rbac" pod "fe024e78-834e-450a-99b6-d40ac786e0d4" (UID: "fe024e78-834e-450a-99b6-d40ac786e0d4"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.357938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.357907 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htkb7\" (UniqueName: \"kubernetes.io/projected/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-kube-api-access-htkb7\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.357938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.357933 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq9kc\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-kube-api-access-kq9kc\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.357938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.357943 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.357954 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.357963 5014 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-tenants\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.357973 5014 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-test-loki-lifecycle-secret\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358004 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f16319de-02a0-44df-9fbf-4a175d4ad3b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358019 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdj6v\" (UniqueName: \"kubernetes.io/projected/0efed69a-48e3-4558-8e39-15798caf4908-kube-api-access-vdj6v\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358030 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-grpc\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358039 5014 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/fe024e78-834e-450a-99b6-d40ac786e0d4-tls-secret\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358048 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/fe024e78-834e-450a-99b6-d40ac786e0d4-data-science-usage-logs-lokistack-gateway-client-http\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358057 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358065 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358078 5014 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-rbac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358092 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-grpc\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358106 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc-data-science-usage-logs-lokistack-ingester-http\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358118 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/0efed69a-48e3-4558-8e39-15798caf4908-data-science-usage-logs-lokistack-distributor-http\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.358157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358148 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9bd50dcf-f652-43c4-bf16-dd7f49cb7f7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:53:48.358691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.358163 5014 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/fe024e78-834e-450a-99b6-d40ac786e0d4-lokistack-gateway\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.368507 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.368490 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-f16319de-02a0-44df-9fbf-4a175d4ad3b9" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:53:48.373196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.373179 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-9bd50dcf-f652-43c4-bf16-dd7f49cb7f7e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:53:48.459198 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.459144 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-9bd50dcf-f652-43c4-bf16-dd7f49cb7f7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b132b0a533702b7\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-7414cac91b824b424fe894433d626f257c573c32523e49fe1ee6ffe352546eec\"" Oct 30 18:53:48.459198 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.459165 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-f16319de-02a0-44df-9fbf-4a175d4ad3b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-026b4460e5abf51ac\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-2d8959d2a75b9a714945a32be7289b244f56b71780476fc260ae0eeed79f1d91\"" Oct 30 18:53:48.705060 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.705027 5014 generic.go:356] "Generic (PLEG): container finished" podID="0efed69a-48e3-4558-8e39-15798caf4908" containerID="71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c" exitCode=0 Oct 30 18:53:48.705254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.705081 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" Oct 30 18:53:48.705254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.705111 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" event={"ID":"0efed69a-48e3-4558-8e39-15798caf4908","Type":"ContainerDied","Data":"71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c"} Oct 30 18:53:48.705254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.705161 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4" event={"ID":"0efed69a-48e3-4558-8e39-15798caf4908","Type":"ContainerDied","Data":"a6e1e4fe99fdec1717b1010672efff0f2fc7774b0e4d700e07648d22b9a660d4"} Oct 30 18:53:48.705254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.705185 5014 scope.go:122] "RemoveContainer" containerID="71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c" Oct 30 18:53:48.706486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.706463 5014 generic.go:356] "Generic (PLEG): container finished" podID="583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" containerID="1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f" exitCode=0 Oct 30 18:53:48.706612 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.706519 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 18:53:48.706612 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.706537 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc","Type":"ContainerDied","Data":"1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f"} Oct 30 18:53:48.706612 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.706571 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"583a0cd2-a73f-4ecf-9949-5a1cf9d659bc","Type":"ContainerDied","Data":"1f219ef3d568d01fd78e62044ef05ae9dcb2caf738f7239893427d2c27d870f5"} Oct 30 18:53:48.708450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.708428 5014 generic.go:356] "Generic (PLEG): container finished" podID="fe024e78-834e-450a-99b6-d40ac786e0d4" containerID="f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1" exitCode=0 Oct 30 18:53:48.708628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.708463 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" event={"ID":"fe024e78-834e-450a-99b6-d40ac786e0d4","Type":"ContainerDied","Data":"f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1"} Oct 30 18:53:48.708628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.708480 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" Oct 30 18:53:48.708628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.708488 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr" event={"ID":"fe024e78-834e-450a-99b6-d40ac786e0d4","Type":"ContainerDied","Data":"bf298176720dc7210ba939b63caf897fc9b3f9ded8e14ce9167cd1ffc9228eb8"} Oct 30 18:53:48.716215 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.716199 5014 scope.go:122] "RemoveContainer" containerID="71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c" Oct 30 18:53:48.716494 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:53:48.716472 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c\": container with ID starting with 71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c not found: ID does not exist" containerID="71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c" Oct 30 18:53:48.716644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.716499 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c"} err="failed to get container status \"71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c\": rpc error: code = NotFound desc = could not find container \"71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c\": container with ID starting with 71238d73c0154e53aab17b7d48e5fef96574cae41d528e2352d30ace2f3ae35c not found: ID does not exist" Oct 30 18:53:48.716644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.716521 5014 scope.go:122] "RemoveContainer" containerID="1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f" Oct 30 18:53:48.726067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.726051 5014 scope.go:122] "RemoveContainer" containerID="1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f" Oct 30 18:53:48.726322 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:53:48.726305 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f\": container with ID starting with 1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f not found: ID does not exist" containerID="1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f" Oct 30 18:53:48.726372 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.726328 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f"} err="failed to get container status \"1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f\": rpc error: code = NotFound desc = could not find container \"1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f\": container with ID starting with 1a4f53f93ae738e2020159baaad20ebf10bb5740a6f5b0d86a3bd21815c4608f not found: ID does not exist" Oct 30 18:53:48.726372 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.726342 5014 scope.go:122] "RemoveContainer" containerID="edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc" Oct 30 18:53:48.733559 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.733544 5014 scope.go:122] "RemoveContainer" containerID="f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1" Oct 30 18:53:48.741235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.741219 5014 scope.go:122] "RemoveContainer" containerID="edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc" Oct 30 18:53:48.741507 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:53:48.741489 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc\": container with ID starting with edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc not found: ID does not exist" containerID="edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc" Oct 30 18:53:48.741553 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.741512 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc"} err="failed to get container status \"edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc\": rpc error: code = NotFound desc = could not find container \"edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc\": container with ID starting with edc7e93e309871d1c64ffb4b0b08d346e8ab4b1b554dd936f0e64802860c1fcc not found: ID does not exist" Oct 30 18:53:48.741553 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.741527 5014 scope.go:122] "RemoveContainer" containerID="f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1" Oct 30 18:53:48.741808 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:53:48.741791 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1\": container with ID starting with f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1 not found: ID does not exist" containerID="f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1" Oct 30 18:53:48.741853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.741813 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1"} err="failed to get container status \"f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1\": rpc error: code = NotFound desc = could not find container \"f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1\": container with ID starting with f9078a6322d0c8118c9905ef33bb65446a671014a3a956d62e37ffcedf1008f1 not found: ID does not exist" Oct 30 18:53:48.755172 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.755153 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4"] Oct 30 18:53:48.766860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.766840 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-l5pz4"] Oct 30 18:53:48.829100 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.829078 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 18:53:48.835555 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.835535 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 18:53:48.869337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.869312 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr"] Oct 30 18:53:48.873719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:48.873700 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-95pwr"] Oct 30 18:53:49.140393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:49.140362 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0efed69a-48e3-4558-8e39-15798caf4908" path="/var/lib/kubelet/pods/0efed69a-48e3-4558-8e39-15798caf4908/volumes" Oct 30 18:53:49.140945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:49.140926 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" path="/var/lib/kubelet/pods/583a0cd2-a73f-4ecf-9949-5a1cf9d659bc/volumes" Oct 30 18:53:49.145135 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:49.141726 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f359add-07be-44c3-8214-1942f5846ae7" path="/var/lib/kubelet/pods/5f359add-07be-44c3-8214-1942f5846ae7/volumes" Oct 30 18:53:49.145135 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:49.142680 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4f2111-4745-41bf-a1f7-a1fe6217260c" path="/var/lib/kubelet/pods/8c4f2111-4745-41bf-a1f7-a1fe6217260c/volumes" Oct 30 18:53:49.145135 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:53:49.143207 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe024e78-834e-450a-99b6-d40ac786e0d4" path="/var/lib/kubelet/pods/fe024e78-834e-450a-99b6-d40ac786e0d4/volumes" Oct 30 18:54:13.332805 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:54:13.332719 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:54:21.510446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510411 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr"] Oct 30 18:54:21.510863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510738 5014 cpu_manager.go:414] "containerMap: removing container" podUID="583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" containerName="loki-ingester" Oct 30 18:54:21.510863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510750 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" containerName="loki-ingester" Oct 30 18:54:21.510863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510760 5014 cpu_manager.go:414] "containerMap: removing container" podUID="fe024e78-834e-450a-99b6-d40ac786e0d4" containerName="gateway" Oct 30 18:54:21.510863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510765 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe024e78-834e-450a-99b6-d40ac786e0d4" containerName="gateway" Oct 30 18:54:21.510863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510784 5014 cpu_manager.go:414] "containerMap: removing container" podUID="0efed69a-48e3-4558-8e39-15798caf4908" containerName="loki-distributor" Oct 30 18:54:21.510863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510789 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0efed69a-48e3-4558-8e39-15798caf4908" containerName="loki-distributor" Oct 30 18:54:21.510863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510799 5014 cpu_manager.go:414] "containerMap: removing container" podUID="8c4f2111-4745-41bf-a1f7-a1fe6217260c" containerName="loki-querier" Oct 30 18:54:21.510863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510803 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4f2111-4745-41bf-a1f7-a1fe6217260c" containerName="loki-querier" Oct 30 18:54:21.510863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510821 5014 cpu_manager.go:414] "containerMap: removing container" podUID="fe024e78-834e-450a-99b6-d40ac786e0d4" containerName="opa" Oct 30 18:54:21.510863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510825 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe024e78-834e-450a-99b6-d40ac786e0d4" containerName="opa" Oct 30 18:54:21.511546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510885 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe024e78-834e-450a-99b6-d40ac786e0d4" containerName="opa" Oct 30 18:54:21.511546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510896 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="583a0cd2-a73f-4ecf-9949-5a1cf9d659bc" containerName="loki-ingester" Oct 30 18:54:21.511546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510903 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe024e78-834e-450a-99b6-d40ac786e0d4" containerName="gateway" Oct 30 18:54:21.511546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510976 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0efed69a-48e3-4558-8e39-15798caf4908" containerName="loki-distributor" Oct 30 18:54:21.511546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.510989 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4f2111-4745-41bf-a1f7-a1fe6217260c" containerName="loki-querier" Oct 30 18:54:21.514229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.514213 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.529567 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.529540 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:54:21.529747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.529612 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:54:21.529747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.529661 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 18:54:21.529747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.529669 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 18:54:21.529890 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.529753 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:54:21.529890 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.529796 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:54:21.529890 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.529821 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-zqdwb\"" Oct 30 18:54:21.538082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.538055 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr"] Oct 30 18:54:21.558646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.558623 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w"] Oct 30 18:54:21.562352 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.562334 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.583459 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.583438 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w"] Oct 30 18:54:21.632314 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.632290 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 18:54:21.632314 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.632292 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 18:54:21.632453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.632320 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 18:54:21.633062 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.633042 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-5g5wz\"" Oct 30 18:54:21.702644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.702614 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.702724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.702693 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.702783 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.702741 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/89a16b76-3643-4050-8d72-5eaed55c463a-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.702783 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.702760 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfd7\" (UniqueName: \"kubernetes.io/projected/89a16b76-3643-4050-8d72-5eaed55c463a-kube-api-access-xqfd7\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.702783 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.702779 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.702919 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.702897 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.702961 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.702950 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.702999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.702974 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.703042 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.703031 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.703081 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.703053 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm9hl\" (UniqueName: \"kubernetes.io/projected/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-api-access-sm9hl\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.804079 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.804053 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.804205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.804091 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.804246 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:54:21.804235 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 18:54:21.804308 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.804288 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/89a16b76-3643-4050-8d72-5eaed55c463a-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.804308 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:54:21.804295 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-client-cert podName:414090c8-84f5-4508-aec9-32e2a4ec18e8 nodeName:}" failed. No retries permitted until 2026-10-30 18:54:22.304279037 +0000 UTC m=+3475.292619506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-v28xr" (UID: "414090c8-84f5-4508-aec9-32e2a4ec18e8") : secret "prometheus-operated-tls" not found Oct 30 18:54:21.804437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.804340 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfd7\" (UniqueName: \"kubernetes.io/projected/89a16b76-3643-4050-8d72-5eaed55c463a-kube-api-access-xqfd7\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.804437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.804359 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.804437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.804401 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.804437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.804427 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.804685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.804447 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.804685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.804500 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.804685 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.804528 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9hl\" (UniqueName: \"kubernetes.io/projected/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-api-access-sm9hl\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.804873 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.804839 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.805070 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.805041 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.805360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.805179 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.805360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.805244 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.807078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.807046 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.807078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.807063 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.807210 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.807063 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/89a16b76-3643-4050-8d72-5eaed55c463a-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.827740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.827719 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9hl\" (UniqueName: \"kubernetes.io/projected/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-api-access-sm9hl\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:21.827848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.827831 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfd7\" (UniqueName: \"kubernetes.io/projected/89a16b76-3643-4050-8d72-5eaed55c463a-kube-api-access-xqfd7\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lz28w\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:21.881846 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:21.881822 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:54:22.195964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:22.195890 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w"] Oct 30 18:54:22.201002 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:54:22.200975 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a16b76_3643_4050_8d72_5eaed55c463a.slice/crio-ccba2728027096c3efb9a9f1b6c4ca4ba798ff78fe3601a66a883d1bdd6bb4fb WatchSource:0}: Error finding container ccba2728027096c3efb9a9f1b6c4ca4ba798ff78fe3601a66a883d1bdd6bb4fb: Status 404 returned error can't find the container with id ccba2728027096c3efb9a9f1b6c4ca4ba798ff78fe3601a66a883d1bdd6bb4fb Oct 30 18:54:22.311467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:22.311442 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:22.314026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:22.314005 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v28xr\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:22.428189 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:22.428163 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:22.725632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:22.725531 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr"] Oct 30 18:54:22.729863 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:54:22.729835 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414090c8_84f5_4508_aec9_32e2a4ec18e8.slice/crio-924355ce619c1278a48cc6bde8f2e600a78272556c9e28cc786269d67bdf8c84 WatchSource:0}: Error finding container 924355ce619c1278a48cc6bde8f2e600a78272556c9e28cc786269d67bdf8c84: Status 404 returned error can't find the container with id 924355ce619c1278a48cc6bde8f2e600a78272556c9e28cc786269d67bdf8c84 Oct 30 18:54:22.869194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:22.869159 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" event={"ID":"414090c8-84f5-4508-aec9-32e2a4ec18e8","Type":"ContainerStarted","Data":"924355ce619c1278a48cc6bde8f2e600a78272556c9e28cc786269d67bdf8c84"} Oct 30 18:54:22.871385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:22.871355 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" event={"ID":"89a16b76-3643-4050-8d72-5eaed55c463a","Type":"ContainerStarted","Data":"a275ea43398788de44b6f3f13cd152cb32984ce89cfb5473062c9e676d7b0e18"} Oct 30 18:54:22.871500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:22.871431 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" event={"ID":"89a16b76-3643-4050-8d72-5eaed55c463a","Type":"ContainerStarted","Data":"856f15d1dc4552340e63e083d84e13e26b08c912cd4072dc0dde95caf6f99291"} Oct 30 18:54:22.871560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:22.871498 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" event={"ID":"89a16b76-3643-4050-8d72-5eaed55c463a","Type":"ContainerStarted","Data":"6478f76e798279e0d65a96f9918e2487d4b94573dc7368fea271968aa02ce2c1"} Oct 30 18:54:22.871560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:22.871536 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" event={"ID":"89a16b76-3643-4050-8d72-5eaed55c463a","Type":"ContainerStarted","Data":"ccba2728027096c3efb9a9f1b6c4ca4ba798ff78fe3601a66a883d1bdd6bb4fb"} Oct 30 18:54:22.913180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:22.913076 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" podStartSLOduration=1.91306201 podStartE2EDuration="1.91306201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:22.912399011 +0000 UTC m=+3475.900739499" watchObservedRunningTime="2026-10-30 18:54:22.91306201 +0000 UTC m=+3475.901402497" Oct 30 18:54:23.163319 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.163291 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:54:23.168447 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.168425 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.183839 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.183790 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:54:23.183839 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.183790 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:54:23.183839 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.183825 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:54:23.184103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.183833 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cnw9r\"" Oct 30 18:54:23.184103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.183858 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:54:23.184103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.183925 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:54:23.196658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.196618 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:54:23.218529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.218500 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfnqv\" (UniqueName: \"kubernetes.io/projected/68531c1c-f61b-4212-8023-20706ffd8fa9-kube-api-access-vfnqv\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.218699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.218570 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/68531c1c-f61b-4212-8023-20706ffd8fa9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.218699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.218673 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.218838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.218717 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.218911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.218826 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.218911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.218892 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.319878 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.319831 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.320062 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.319936 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.320062 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.319963 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.320062 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:54:23.320025 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 30 18:54:23.320241 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:54:23.320092 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-otlp-tls-certs podName:68531c1c-f61b-4212-8023-20706ffd8fa9 nodeName:}" failed. No retries permitted until 2026-10-30 18:54:23.820074889 +0000 UTC m=+3476.808415374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "68531c1c-f61b-4212-8023-20706ffd8fa9") : secret "data-science-collector-collector-headless-tls" not found Oct 30 18:54:23.320241 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.320110 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfnqv\" (UniqueName: \"kubernetes.io/projected/68531c1c-f61b-4212-8023-20706ffd8fa9-kube-api-access-vfnqv\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.320241 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.320155 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/68531c1c-f61b-4212-8023-20706ffd8fa9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.320241 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.320185 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.321875 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.321475 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/68531c1c-f61b-4212-8023-20706ffd8fa9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.324456 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.323490 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.324899 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.324877 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.326062 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.326040 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.348046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.347994 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfnqv\" (UniqueName: \"kubernetes.io/projected/68531c1c-f61b-4212-8023-20706ffd8fa9-kube-api-access-vfnqv\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.825149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.825121 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.827745 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.827722 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:23.877778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.877743 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" event={"ID":"414090c8-84f5-4508-aec9-32e2a4ec18e8","Type":"ContainerStarted","Data":"a4ca2ad736f644d8948bb8fef1c4f8dd72c37980b2c4bcb7e1b453fecc1385c1"} Oct 30 18:54:23.878018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.877998 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:23.918955 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:23.918911 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" podStartSLOduration=2.918899464 podStartE2EDuration="2.918899464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:23.91774125 +0000 UTC m=+3476.906081736" watchObservedRunningTime="2026-10-30 18:54:23.918899464 +0000 UTC m=+3476.907239951" Oct 30 18:54:24.085706 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:24.085649 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:54:24.382380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:24.382300 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:54:24.387104 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:54:24.387076 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68531c1c_f61b_4212_8023_20706ffd8fa9.slice/crio-86e2dbd3bd8d34c1de2d70c50f828a6a646c8da8df8948164b062049b7ef0ab9 WatchSource:0}: Error finding container 86e2dbd3bd8d34c1de2d70c50f828a6a646c8da8df8948164b062049b7ef0ab9: Status 404 returned error can't find the container with id 86e2dbd3bd8d34c1de2d70c50f828a6a646c8da8df8948164b062049b7ef0ab9 Oct 30 18:54:24.389883 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:24.389849 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:54:24.389954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:24.389913 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:54:24.884724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:24.884685 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"68531c1c-f61b-4212-8023-20706ffd8fa9","Type":"ContainerStarted","Data":"e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c"} Oct 30 18:54:24.885141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:24.884746 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"68531c1c-f61b-4212-8023-20706ffd8fa9","Type":"ContainerStarted","Data":"86e2dbd3bd8d34c1de2d70c50f828a6a646c8da8df8948164b062049b7ef0ab9"} Oct 30 18:54:24.927073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:24.927026 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.927013551 podStartE2EDuration="1.927013551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:24.925692156 +0000 UTC m=+3477.914032643" watchObservedRunningTime="2026-10-30 18:54:24.927013551 +0000 UTC m=+3477.915354038" Oct 30 18:54:29.886988 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:29.886959 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:54:30.103624 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.103578 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l"] Oct 30 18:54:30.107180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.107162 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" Oct 30 18:54:30.121986 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.121964 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-5mtd6\"" Oct 30 18:54:30.127294 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.127271 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l"] Oct 30 18:54:30.174973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.174916 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlkj\" (UniqueName: \"kubernetes.io/projected/63258791-442a-4c75-9ecd-1cb65a8f0661-kube-api-access-xnlkj\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l\" (UID: \"63258791-442a-4c75-9ecd-1cb65a8f0661\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" Oct 30 18:54:30.275650 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.275620 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlkj\" (UniqueName: \"kubernetes.io/projected/63258791-442a-4c75-9ecd-1cb65a8f0661-kube-api-access-xnlkj\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l\" (UID: \"63258791-442a-4c75-9ecd-1cb65a8f0661\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" Oct 30 18:54:30.298832 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.298806 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlkj\" (UniqueName: \"kubernetes.io/projected/63258791-442a-4c75-9ecd-1cb65a8f0661-kube-api-access-xnlkj\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l\" (UID: \"63258791-442a-4c75-9ecd-1cb65a8f0661\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" Oct 30 18:54:30.429727 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.429676 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" Oct 30 18:54:30.726972 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.726901 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l"] Oct 30 18:54:30.731694 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:54:30.731656 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63258791_442a_4c75_9ecd_1cb65a8f0661.slice/crio-9c03635c86bb6a61c22e04bce0a72bb1b59abc63861a4a3fd4f089ba96390661 WatchSource:0}: Error finding container 9c03635c86bb6a61c22e04bce0a72bb1b59abc63861a4a3fd4f089ba96390661: Status 404 returned error can't find the container with id 9c03635c86bb6a61c22e04bce0a72bb1b59abc63861a4a3fd4f089ba96390661 Oct 30 18:54:30.922654 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.922618 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" event={"ID":"63258791-442a-4c75-9ecd-1cb65a8f0661","Type":"ContainerStarted","Data":"eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad"} Oct 30 18:54:30.922654 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.922658 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" event={"ID":"63258791-442a-4c75-9ecd-1cb65a8f0661","Type":"ContainerStarted","Data":"9c03635c86bb6a61c22e04bce0a72bb1b59abc63861a4a3fd4f089ba96390661"} Oct 30 18:54:30.952035 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.952006 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:54:30.956627 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.956576 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:30.964151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.964100 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" podStartSLOduration=0.964084809 podStartE2EDuration="964.084809ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:30.962754427 +0000 UTC m=+3483.951094916" watchObservedRunningTime="2026-10-30 18:54:30.964084809 +0000 UTC m=+3483.952425295" Oct 30 18:54:30.982534 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.982471 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:30.982696 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.982539 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzjb\" (UniqueName: \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-kube-api-access-mnzjb\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:30.982696 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.982587 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:30.982696 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.982642 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:30.982877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.982751 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:30.982877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.982810 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:30.982877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.982842 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:30.996064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:30.996036 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:54:31.041587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.041532 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-8dld8\"" Oct 30 18:54:31.041587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.041548 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:54:31.041587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.041532 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:54:31.041825 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.041612 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:54:31.041825 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.041633 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:54:31.045390 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.045366 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:54:31.050061 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.050045 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.077184 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.077164 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:54:31.083951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.083924 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.084054 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.084022 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.084125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.084074 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.084125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.084095 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.084218 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.084191 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.084275 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.084234 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzjb\" (UniqueName: \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-kube-api-access-mnzjb\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.084342 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.084269 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.085285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.085261 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.086779 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.086762 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.086886 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.086835 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.087201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.087176 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.087201 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.087194 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.087658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.087638 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.107634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.107588 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzjb\" (UniqueName: \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-kube-api-access-mnzjb\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.135448 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.135426 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:54:31.135567 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.135426 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:54:31.135666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.135566 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:54:31.141741 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.141723 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:54:31.141840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.141828 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:54:31.141901 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.141842 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:54:31.141901 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.141843 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:54:31.141973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.141916 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-b7kx8\"" Oct 30 18:54:31.141973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.141953 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:54:31.185333 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185310 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.185405 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185345 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.185405 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185378 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.185483 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185420 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.185483 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185467 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.185546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185533 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.185580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185558 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.185643 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185615 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.185696 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185683 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.185732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185705 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.185764 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185749 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgncs\" (UniqueName: \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-kube-api-access-pgncs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.185838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.185821 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.277060 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.277005 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:31.286425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.286405 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.286520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.286433 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.286520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.286450 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.286520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.286467 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.286778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.286751 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.286884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.286850 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.286977 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.286909 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.286977 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.286938 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.286977 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.286972 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgncs\" (UniqueName: \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-kube-api-access-pgncs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.287141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.287017 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.287141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.287056 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.287141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.287100 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.287472 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.287371 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.288188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.288163 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.288563 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.288534 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.290272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.290236 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.290712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.290691 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.290892 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.290749 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.291899 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.291875 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.292139 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.292048 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.292139 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.292101 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.292444 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.292419 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.292555 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.292509 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.310253 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.310220 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgncs\" (UniqueName: \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-kube-api-access-pgncs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:31.609319 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.609282 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:54:31.613323 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:54:31.613297 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350afcca_4456_43a6_9087_3f8decb7942d.slice/crio-8b05690a577e499db65b28afb52799d5d42d344d2ab7bd4ce3c4cca126786379 WatchSource:0}: Error finding container 8b05690a577e499db65b28afb52799d5d42d344d2ab7bd4ce3c4cca126786379: Status 404 returned error can't find the container with id 8b05690a577e499db65b28afb52799d5d42d344d2ab7bd4ce3c4cca126786379 Oct 30 18:54:31.929294 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.929267 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"350afcca-4456-43a6-9087-3f8decb7942d","Type":"ContainerStarted","Data":"5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9"} Oct 30 18:54:31.929699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:31.929320 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"350afcca-4456-43a6-9087-3f8decb7942d","Type":"ContainerStarted","Data":"8b05690a577e499db65b28afb52799d5d42d344d2ab7bd4ce3c4cca126786379"} Oct 30 18:54:33.506798 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:33.506731 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:33.535207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:33.535181 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:33.607312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:33.607290 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:33.607444 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:33.607424 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:33.621305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:33.621279 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:33.723544 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:33.723515 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/26c702426f1aeab2e9e6116e994cbd362b1230f690bb6d32002e36ee94fa3aec/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:33.749914 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:33.749893 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:33.770611 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:33.770548 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:34.101908 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:34.101875 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:54:34.105615 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:54:34.105574 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0496e67_dcde_4d93_a18b_f23ac2b9d649.slice/crio-3940abd5f07d126d4cdb7ddf8749e1aaff6bb98cc07cfc5fbf5f14ea4a4376cb WatchSource:0}: Error finding container 3940abd5f07d126d4cdb7ddf8749e1aaff6bb98cc07cfc5fbf5f14ea4a4376cb: Status 404 returned error can't find the container with id 3940abd5f07d126d4cdb7ddf8749e1aaff6bb98cc07cfc5fbf5f14ea4a4376cb Oct 30 18:54:34.946700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:34.946668 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d0496e67-dcde-4d93-a18b-f23ac2b9d649","Type":"ContainerStarted","Data":"40619b7f2017e56c8555efb5a0ef9490643cc9053204fa60989dcbb66dd15a0b"} Oct 30 18:54:34.947069 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:34.946754 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d0496e67-dcde-4d93-a18b-f23ac2b9d649","Type":"ContainerStarted","Data":"3940abd5f07d126d4cdb7ddf8749e1aaff6bb98cc07cfc5fbf5f14ea4a4376cb"} Oct 30 18:54:35.952990 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:35.952956 5014 generic.go:356] "Generic (PLEG): container finished" podID="350afcca-4456-43a6-9087-3f8decb7942d" containerID="5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9" exitCode=0 Oct 30 18:54:35.953456 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:35.953037 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"350afcca-4456-43a6-9087-3f8decb7942d","Type":"ContainerDied","Data":"5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9"} Oct 30 18:54:36.959492 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:36.959461 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"350afcca-4456-43a6-9087-3f8decb7942d","Type":"ContainerStarted","Data":"6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72"} Oct 30 18:54:36.959852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:36.959522 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"350afcca-4456-43a6-9087-3f8decb7942d","Type":"ContainerStarted","Data":"cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86"} Oct 30 18:54:36.959852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:36.959724 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:37.028656 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:37.028614 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.544230753 podStartE2EDuration="7.028580145s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484349392s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:37.017144401 +0000 UTC m=+3490.005484887" watchObservedRunningTime="2026-10-30 18:54:37.028580145 +0000 UTC m=+3490.016920631" Oct 30 18:54:38.971258 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:38.971224 5014 generic.go:356] "Generic (PLEG): container finished" podID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerID="40619b7f2017e56c8555efb5a0ef9490643cc9053204fa60989dcbb66dd15a0b" exitCode=0 Oct 30 18:54:38.971258 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:38.971261 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d0496e67-dcde-4d93-a18b-f23ac2b9d649","Type":"ContainerDied","Data":"40619b7f2017e56c8555efb5a0ef9490643cc9053204fa60989dcbb66dd15a0b"} Oct 30 18:54:39.978474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:39.978402 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d0496e67-dcde-4d93-a18b-f23ac2b9d649","Type":"ContainerStarted","Data":"20d3638e6ddb179f57ecef3d126f166650e80f5fbac4c72d9e876c856e31320a"} Oct 30 18:54:39.978474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:39.978461 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d0496e67-dcde-4d93-a18b-f23ac2b9d649","Type":"ContainerStarted","Data":"c8c2de9b45f8cf7babf47286f9d019f75586b66b5f846f0e925d102343fba5b2"} Oct 30 18:54:39.978948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:39.978487 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d0496e67-dcde-4d93-a18b-f23ac2b9d649","Type":"ContainerStarted","Data":"6d2225d6fd181898cf7485359fbf42450e801f3a0060dbd9c39b495cc975f449"} Oct 30 18:54:40.053990 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:40.053946 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=5.36870344 podStartE2EDuration="9.053933399s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.685229959s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:40.041222788 +0000 UTC m=+3493.029563276" watchObservedRunningTime="2026-10-30 18:54:40.053933399 +0000 UTC m=+3493.042273894" Oct 30 18:54:42.966468 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:42.966440 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:54:43.331301 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:54:43.331270 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:54:43.771629 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:43.771535 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:48.770887 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:48.770850 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:48.776322 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:48.776303 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:54:49.026766 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:54:49.026700 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:07.450833 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:07.450789 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:55:07.451309 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:07.451077 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="prometheus" containerID="cri-o://6d2225d6fd181898cf7485359fbf42450e801f3a0060dbd9c39b495cc975f449" gracePeriod=600 Oct 30 18:55:07.451309 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:07.451126 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="thanos-sidecar" containerID="cri-o://20d3638e6ddb179f57ecef3d126f166650e80f5fbac4c72d9e876c856e31320a" gracePeriod=600 Oct 30 18:55:07.451309 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:07.451147 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="config-reloader" containerID="cri-o://c8c2de9b45f8cf7babf47286f9d019f75586b66b5f846f0e925d102343fba5b2" gracePeriod=600 Oct 30 18:55:08.121863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.121831 5014 generic.go:356] "Generic (PLEG): container finished" podID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerID="20d3638e6ddb179f57ecef3d126f166650e80f5fbac4c72d9e876c856e31320a" exitCode=0 Oct 30 18:55:08.121863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.121856 5014 generic.go:356] "Generic (PLEG): container finished" podID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerID="c8c2de9b45f8cf7babf47286f9d019f75586b66b5f846f0e925d102343fba5b2" exitCode=0 Oct 30 18:55:08.121863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.121863 5014 generic.go:356] "Generic (PLEG): container finished" podID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerID="6d2225d6fd181898cf7485359fbf42450e801f3a0060dbd9c39b495cc975f449" exitCode=0 Oct 30 18:55:08.122168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.121898 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d0496e67-dcde-4d93-a18b-f23ac2b9d649","Type":"ContainerDied","Data":"20d3638e6ddb179f57ecef3d126f166650e80f5fbac4c72d9e876c856e31320a"} Oct 30 18:55:08.122168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.121959 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d0496e67-dcde-4d93-a18b-f23ac2b9d649","Type":"ContainerDied","Data":"c8c2de9b45f8cf7babf47286f9d019f75586b66b5f846f0e925d102343fba5b2"} Oct 30 18:55:08.122168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.121985 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d0496e67-dcde-4d93-a18b-f23ac2b9d649","Type":"ContainerDied","Data":"6d2225d6fd181898cf7485359fbf42450e801f3a0060dbd9c39b495cc975f449"} Oct 30 18:55:08.218827 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.218805 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:08.288104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288078 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.288242 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288115 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-thanos-prometheus-http-client-file\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.288242 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288175 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.288242 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288231 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.288543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288521 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288743 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-0" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288759 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config-out\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288794 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-1" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288833 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-kube-api-access-pgncs\" (UniqueName: \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-kube-api-access-pgncs\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288874 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-secret-prometheus-web-tls-ca\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288913 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288947 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-tls-assets\" (UniqueName: \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-tls-assets\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.288973 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.289114 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.289203 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config\") pod \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\" (UID: \"d0496e67-dcde-4d93-a18b-f23ac2b9d649\") " Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.289524 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.289855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.289547 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.290711 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.290187 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-2" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:55:08.291494 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.291468 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-thanos-prometheus-http-client-file" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:08.291750 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.291726 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-kube-api-access-pgncs" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "kube-api-access-pgncs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:55:08.292215 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.292188 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:08.292529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.292503 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config-out" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:55:08.293876 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.293848 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:08.293957 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.293869 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-tls-assets" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:55:08.294428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.294409 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-secret-prometheus-web-tls-ca" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:08.295217 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.295194 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "pvc-c02e8472-bb1a-4974-aa89-86eef715889b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:55:08.295311 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.295285 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:08.306728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.306707 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config" pod "d0496e67-dcde-4d93-a18b-f23ac2b9d649" (UID: "d0496e67-dcde-4d93-a18b-f23ac2b9d649"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:08.390336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.390270 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:55:08.390336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.390306 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.390336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.390316 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgncs\" (UniqueName: \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-kube-api-access-pgncs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.390336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.390325 5014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.390336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.390336 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d0496e67-dcde-4d93-a18b-f23ac2b9d649-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.390336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.390345 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0496e67-dcde-4d93-a18b-f23ac2b9d649-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.390617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.390353 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.390617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.390361 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.390617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.390369 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.390617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.390379 5014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.390617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.390388 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d0496e67-dcde-4d93-a18b-f23ac2b9d649-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:08.401988 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.401972 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c02e8472-bb1a-4974-aa89-86eef715889b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:55:08.491152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:08.491128 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" Oct 30 18:55:09.130784 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:09.130753 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d0496e67-dcde-4d93-a18b-f23ac2b9d649","Type":"ContainerDied","Data":"3940abd5f07d126d4cdb7ddf8749e1aaff6bb98cc07cfc5fbf5f14ea4a4376cb"} Oct 30 18:55:09.130784 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:09.130778 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:09.130989 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:09.130832 5014 scope.go:122] "RemoveContainer" containerID="20d3638e6ddb179f57ecef3d126f166650e80f5fbac4c72d9e876c856e31320a" Oct 30 18:55:09.142040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:09.142017 5014 scope.go:122] "RemoveContainer" containerID="c8c2de9b45f8cf7babf47286f9d019f75586b66b5f846f0e925d102343fba5b2" Oct 30 18:55:09.151449 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:09.151423 5014 scope.go:122] "RemoveContainer" containerID="6d2225d6fd181898cf7485359fbf42450e801f3a0060dbd9c39b495cc975f449" Oct 30 18:55:09.161467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:09.161439 5014 scope.go:122] "RemoveContainer" containerID="40619b7f2017e56c8555efb5a0ef9490643cc9053204fa60989dcbb66dd15a0b" Oct 30 18:55:09.196064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:09.196039 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:55:09.207560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:09.207538 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:55:11.111018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:11.110978 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:11.111485 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:11.111215 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="68531c1c-f61b-4212-8023-20706ffd8fa9" containerName="otc-container" containerID="cri-o://e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c" gracePeriod=30 Oct 30 18:55:11.142667 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:11.142637 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" path="/var/lib/kubelet/pods/d0496e67-dcde-4d93-a18b-f23ac2b9d649/volumes" Oct 30 18:55:13.333211 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:55:13.333180 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:55:13.457146 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.457125 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:13.537084 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.537024 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-monitoring-tls-certs\") pod \"68531c1c-f61b-4212-8023-20706ffd8fa9\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " Oct 30 18:55:13.537084 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.537071 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68531c1c-f61b-4212-8023-20706ffd8fa9-kube-api-access-vfnqv\" (UniqueName: \"kubernetes.io/projected/68531c1c-f61b-4212-8023-20706ffd8fa9-kube-api-access-vfnqv\") pod \"68531c1c-f61b-4212-8023-20706ffd8fa9\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " Oct 30 18:55:13.537237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.537110 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-data-science-collector-ta-client-cert\") pod \"68531c1c-f61b-4212-8023-20706ffd8fa9\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " Oct 30 18:55:13.537237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.537142 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-tls-certs\") pod \"68531c1c-f61b-4212-8023-20706ffd8fa9\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " Oct 30 18:55:13.537237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.537221 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68531c1c-f61b-4212-8023-20706ffd8fa9-otc-internal\" (UniqueName: \"kubernetes.io/configmap/68531c1c-f61b-4212-8023-20706ffd8fa9-otc-internal\") pod \"68531c1c-f61b-4212-8023-20706ffd8fa9\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " Oct 30 18:55:13.537364 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.537285 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-otlp-tls-certs\") pod \"68531c1c-f61b-4212-8023-20706ffd8fa9\" (UID: \"68531c1c-f61b-4212-8023-20706ffd8fa9\") " Oct 30 18:55:13.537818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.537709 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68531c1c-f61b-4212-8023-20706ffd8fa9-otc-internal" pod "68531c1c-f61b-4212-8023-20706ffd8fa9" (UID: "68531c1c-f61b-4212-8023-20706ffd8fa9"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:55:13.539335 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.539314 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-monitoring-tls-certs" pod "68531c1c-f61b-4212-8023-20706ffd8fa9" (UID: "68531c1c-f61b-4212-8023-20706ffd8fa9"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:13.539439 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.539403 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-data-science-collector-ta-client-cert" pod "68531c1c-f61b-4212-8023-20706ffd8fa9" (UID: "68531c1c-f61b-4212-8023-20706ffd8fa9"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:13.539517 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.539495 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-otlp-tls-certs" pod "68531c1c-f61b-4212-8023-20706ffd8fa9" (UID: "68531c1c-f61b-4212-8023-20706ffd8fa9"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:13.539828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.539809 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-tls-certs" pod "68531c1c-f61b-4212-8023-20706ffd8fa9" (UID: "68531c1c-f61b-4212-8023-20706ffd8fa9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:13.539881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.539825 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68531c1c-f61b-4212-8023-20706ffd8fa9-kube-api-access-vfnqv" pod "68531c1c-f61b-4212-8023-20706ffd8fa9" (UID: "68531c1c-f61b-4212-8023-20706ffd8fa9"). InnerVolumeSpecName "kube-api-access-vfnqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:55:13.637925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.637897 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:13.637925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.637919 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfnqv\" (UniqueName: \"kubernetes.io/projected/68531c1c-f61b-4212-8023-20706ffd8fa9-kube-api-access-vfnqv\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:13.637925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.637928 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:13.638106 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.637937 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:13.638106 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.637946 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/68531c1c-f61b-4212-8023-20706ffd8fa9-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:13.638106 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:13.637953 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/68531c1c-f61b-4212-8023-20706ffd8fa9-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:14.157983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.157951 5014 generic.go:356] "Generic (PLEG): container finished" podID="68531c1c-f61b-4212-8023-20706ffd8fa9" containerID="e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c" exitCode=0 Oct 30 18:55:14.158149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.158000 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.158149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.158018 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"68531c1c-f61b-4212-8023-20706ffd8fa9","Type":"ContainerDied","Data":"e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c"} Oct 30 18:55:14.158149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.158069 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"68531c1c-f61b-4212-8023-20706ffd8fa9","Type":"ContainerDied","Data":"86e2dbd3bd8d34c1de2d70c50f828a6a646c8da8df8948164b062049b7ef0ab9"} Oct 30 18:55:14.158149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.158108 5014 scope.go:122] "RemoveContainer" containerID="e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c" Oct 30 18:55:14.167793 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.167772 5014 scope.go:122] "RemoveContainer" containerID="e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c" Oct 30 18:55:14.168077 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:55:14.168059 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c\": container with ID starting with e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c not found: ID does not exist" containerID="e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c" Oct 30 18:55:14.168124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.168085 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c"} err="failed to get container status \"e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c\": rpc error: code = NotFound desc = could not find container \"e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c\": container with ID starting with e9e4f51b4a49c4d68e5bb6897ab21a3d75b801574d1e4164d9875c630fbef04c not found: ID does not exist" Oct 30 18:55:14.208249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.208227 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:14.219742 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.219717 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:14.250456 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250434 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:14.250838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250823 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="init-config-reloader" Oct 30 18:55:14.250886 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250838 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="init-config-reloader" Oct 30 18:55:14.250886 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250849 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="prometheus" Oct 30 18:55:14.250886 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250854 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="prometheus" Oct 30 18:55:14.250886 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250878 5014 cpu_manager.go:414] "containerMap: removing container" podUID="68531c1c-f61b-4212-8023-20706ffd8fa9" containerName="otc-container" Oct 30 18:55:14.250886 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250883 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68531c1c-f61b-4212-8023-20706ffd8fa9" containerName="otc-container" Oct 30 18:55:14.251235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250891 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="thanos-sidecar" Oct 30 18:55:14.251235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250897 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="thanos-sidecar" Oct 30 18:55:14.251235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250909 5014 cpu_manager.go:414] "containerMap: removing container" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="config-reloader" Oct 30 18:55:14.251235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250914 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="config-reloader" Oct 30 18:55:14.251235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250969 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="prometheus" Oct 30 18:55:14.251235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250981 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="68531c1c-f61b-4212-8023-20706ffd8fa9" containerName="otc-container" Oct 30 18:55:14.251235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.250991 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="thanos-sidecar" Oct 30 18:55:14.251235 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.251002 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0496e67-dcde-4d93-a18b-f23ac2b9d649" containerName="config-reloader" Oct 30 18:55:14.255328 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.255313 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.269610 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.269572 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:55:14.269798 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.269782 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:55:14.269854 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.269813 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:55:14.269854 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.269813 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:55:14.269960 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.269855 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:55:14.269960 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.269896 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cnw9r\"" Oct 30 18:55:14.282096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.282073 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:14.344579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.344552 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/518d323a-10b1-4f21-966e-8454d6e37a52-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.344928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.344582 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.344928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.344693 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6kt\" (UniqueName: \"kubernetes.io/projected/518d323a-10b1-4f21-966e-8454d6e37a52-kube-api-access-sb6kt\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.344928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.344750 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.344928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.344812 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.344928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.344843 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.445868 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.445818 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6kt\" (UniqueName: \"kubernetes.io/projected/518d323a-10b1-4f21-966e-8454d6e37a52-kube-api-access-sb6kt\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.445868 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.445854 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.446032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.445878 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.446032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.445897 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.446032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.445931 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/518d323a-10b1-4f21-966e-8454d6e37a52-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.446189 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.446102 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.446550 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.446518 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/518d323a-10b1-4f21-966e-8454d6e37a52-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.448648 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.448628 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.448734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.448671 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.448734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.448711 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.448809 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.448748 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.468407 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.468381 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6kt\" (UniqueName: \"kubernetes.io/projected/518d323a-10b1-4f21-966e-8454d6e37a52-kube-api-access-sb6kt\") pod \"data-science-collector-collector-0\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.567417 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.567392 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:14.873118 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.873089 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:14.877548 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:55:14.877512 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518d323a_10b1_4f21_966e_8454d6e37a52.slice/crio-d6690021e93ac27553474a2678532e156b0207e20c0feb46eeda292ac3b53e07 WatchSource:0}: Error finding container d6690021e93ac27553474a2678532e156b0207e20c0feb46eeda292ac3b53e07: Status 404 returned error can't find the container with id d6690021e93ac27553474a2678532e156b0207e20c0feb46eeda292ac3b53e07 Oct 30 18:55:14.879747 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.879718 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:55:14.879839 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:14.879775 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:55:15.142234 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:15.142149 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68531c1c-f61b-4212-8023-20706ffd8fa9" path="/var/lib/kubelet/pods/68531c1c-f61b-4212-8023-20706ffd8fa9/volumes" Oct 30 18:55:15.164604 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:15.164565 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"518d323a-10b1-4f21-966e-8454d6e37a52","Type":"ContainerStarted","Data":"0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138"} Oct 30 18:55:15.164748 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:15.164641 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"518d323a-10b1-4f21-966e-8454d6e37a52","Type":"ContainerStarted","Data":"d6690021e93ac27553474a2678532e156b0207e20c0feb46eeda292ac3b53e07"} Oct 30 18:55:15.206822 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:15.206772 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.206756347 podStartE2EDuration="1.206756347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:15.206058621 +0000 UTC m=+3528.194399109" watchObservedRunningTime="2026-10-30 18:55:15.206756347 +0000 UTC m=+3528.195096834" Oct 30 18:55:24.992827 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:24.992790 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:55:24.998104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:24.998083 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.014177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.014112 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:55:25.014177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.014134 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:55:25.014432 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.014223 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:55:25.014432 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.014304 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:55:25.025721 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.025694 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:55:25.025828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.025740 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:55:25.025828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.025741 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:55:25.026006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.025988 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-b7kx8\"" Oct 30 18:55:25.026006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.025999 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:55:25.027504 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.027487 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:55:25.128907 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.128882 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.128999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.128913 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.128999 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.128943 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.129073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.129032 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.129115 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.129064 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.129115 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.129109 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.129180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.129134 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bq5k\" (UniqueName: \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-kube-api-access-5bq5k\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.129180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.129161 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.129246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.129217 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.129280 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.129263 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.129314 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.129288 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.129348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.129305 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230127 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230267 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230157 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230267 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230181 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230322 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bq5k\" (UniqueName: \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-kube-api-access-5bq5k\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230363 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230492 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230407 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230548 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230443 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230678 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230566 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230678 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230620 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230678 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230663 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230696 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.230855 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.230751 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.231124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.231002 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.231892 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.231346 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.232400 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.232356 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.233627 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.233574 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.233736 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.233686 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.234045 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.234001 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.234155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.234066 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.234155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.234103 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.234351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.234333 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.234449 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.234422 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.234574 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.234557 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:25.252878 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:25.252832 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bq5k\" (UniqueName: \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-kube-api-access-5bq5k\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:27.178750 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.178720 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:27.179241 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.178930 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="518d323a-10b1-4f21-966e-8454d6e37a52" containerName="otc-container" containerID="cri-o://0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138" gracePeriod=30 Oct 30 18:55:27.434558 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.434500 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:27.553236 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.553207 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/518d323a-10b1-4f21-966e-8454d6e37a52-otc-internal\" (UniqueName: \"kubernetes.io/configmap/518d323a-10b1-4f21-966e-8454d6e37a52-otc-internal\") pod \"518d323a-10b1-4f21-966e-8454d6e37a52\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " Oct 30 18:55:27.553399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.553246 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-monitoring-tls-certs\") pod \"518d323a-10b1-4f21-966e-8454d6e37a52\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " Oct 30 18:55:27.553399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.553276 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-tls-certs\") pod \"518d323a-10b1-4f21-966e-8454d6e37a52\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " Oct 30 18:55:27.553399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.553295 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-data-science-collector-ta-client-cert\") pod \"518d323a-10b1-4f21-966e-8454d6e37a52\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " Oct 30 18:55:27.553399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.553323 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-otlp-tls-certs\") pod \"518d323a-10b1-4f21-966e-8454d6e37a52\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " Oct 30 18:55:27.553399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.553393 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518d323a-10b1-4f21-966e-8454d6e37a52-kube-api-access-sb6kt\" (UniqueName: \"kubernetes.io/projected/518d323a-10b1-4f21-966e-8454d6e37a52-kube-api-access-sb6kt\") pod \"518d323a-10b1-4f21-966e-8454d6e37a52\" (UID: \"518d323a-10b1-4f21-966e-8454d6e37a52\") " Oct 30 18:55:27.553753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.553678 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518d323a-10b1-4f21-966e-8454d6e37a52-otc-internal" pod "518d323a-10b1-4f21-966e-8454d6e37a52" (UID: "518d323a-10b1-4f21-966e-8454d6e37a52"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:55:27.555752 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.555725 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-tls-certs" pod "518d323a-10b1-4f21-966e-8454d6e37a52" (UID: "518d323a-10b1-4f21-966e-8454d6e37a52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:27.555752 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.555729 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518d323a-10b1-4f21-966e-8454d6e37a52-kube-api-access-sb6kt" pod "518d323a-10b1-4f21-966e-8454d6e37a52" (UID: "518d323a-10b1-4f21-966e-8454d6e37a52"). InnerVolumeSpecName "kube-api-access-sb6kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:55:27.555943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.555749 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-otlp-tls-certs" pod "518d323a-10b1-4f21-966e-8454d6e37a52" (UID: "518d323a-10b1-4f21-966e-8454d6e37a52"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:27.555993 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.555961 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-data-science-collector-ta-client-cert" pod "518d323a-10b1-4f21-966e-8454d6e37a52" (UID: "518d323a-10b1-4f21-966e-8454d6e37a52"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:27.556036 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.556008 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-monitoring-tls-certs" pod "518d323a-10b1-4f21-966e-8454d6e37a52" (UID: "518d323a-10b1-4f21-966e-8454d6e37a52"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:27.654435 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.654408 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb6kt\" (UniqueName: \"kubernetes.io/projected/518d323a-10b1-4f21-966e-8454d6e37a52-kube-api-access-sb6kt\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:27.654435 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.654433 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/518d323a-10b1-4f21-966e-8454d6e37a52-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:27.654615 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.654442 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:27.654615 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.654451 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:27.654615 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.654463 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:27.654615 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.654473 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/518d323a-10b1-4f21-966e-8454d6e37a52-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:55:27.942327 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:27.942294 5014 scope.go:122] "RemoveContainer" containerID="68d8b4fbf071815d809150349b2e4d86428bac1251b9927928b3bbb3b0a09ed3" Oct 30 18:55:28.232391 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.232319 5014 generic.go:356] "Generic (PLEG): container finished" podID="518d323a-10b1-4f21-966e-8454d6e37a52" containerID="0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138" exitCode=0 Oct 30 18:55:28.232391 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.232376 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.232824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.232403 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"518d323a-10b1-4f21-966e-8454d6e37a52","Type":"ContainerDied","Data":"0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138"} Oct 30 18:55:28.232824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.232453 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"518d323a-10b1-4f21-966e-8454d6e37a52","Type":"ContainerDied","Data":"d6690021e93ac27553474a2678532e156b0207e20c0feb46eeda292ac3b53e07"} Oct 30 18:55:28.232824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.232480 5014 scope.go:122] "RemoveContainer" containerID="0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138" Oct 30 18:55:28.240756 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.240735 5014 scope.go:122] "RemoveContainer" containerID="0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138" Oct 30 18:55:28.241017 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:55:28.240996 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138\": container with ID starting with 0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138 not found: ID does not exist" containerID="0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138" Oct 30 18:55:28.241082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.241024 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138"} err="failed to get container status \"0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138\": rpc error: code = NotFound desc = could not find container \"0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138\": container with ID starting with 0aadce1e7884c7f51b1401898a0110895e78467eae645eb310c2e0cc1ab00138 not found: ID does not exist" Oct 30 18:55:28.281794 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.281770 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:28.293464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.293442 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:28.323692 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.323666 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:28.324188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.324156 5014 cpu_manager.go:414] "containerMap: removing container" podUID="518d323a-10b1-4f21-966e-8454d6e37a52" containerName="otc-container" Oct 30 18:55:28.324188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.324175 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518d323a-10b1-4f21-966e-8454d6e37a52" containerName="otc-container" Oct 30 18:55:28.324364 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.324349 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="518d323a-10b1-4f21-966e-8454d6e37a52" containerName="otc-container" Oct 30 18:55:28.328776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.328760 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.343272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.343244 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:55:28.343272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.343251 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:55:28.343398 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.343334 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:55:28.343436 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.343400 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:55:28.343475 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.343447 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cnw9r\"" Oct 30 18:55:28.343517 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.343495 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:55:28.355858 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.355827 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:28.361348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.361326 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.361448 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.361374 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.361510 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.361469 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbgp\" (UniqueName: \"kubernetes.io/projected/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-kube-api-access-2dbgp\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.361570 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.361538 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.361647 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.361612 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.361728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.361705 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.463018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.462982 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.463191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.463025 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.463191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.463060 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbgp\" (UniqueName: \"kubernetes.io/projected/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-kube-api-access-2dbgp\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.463191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.463086 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.463191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.463113 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.463395 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.463308 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.464016 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.463848 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.465768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.465745 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.465768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.465754 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.465944 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.465876 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.466026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.466005 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.486420 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.486364 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbgp\" (UniqueName: \"kubernetes.io/projected/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-kube-api-access-2dbgp\") pod \"data-science-collector-collector-0\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.640796 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.640775 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:55:28.939200 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.939172 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:55:28.943195 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:55:28.943169 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bbb2f4_aeb6_48f5_bb67_9dab8ac2303c.slice/crio-4a8cb0e56d4e0d6a4bd7bff4a9f43d79e4355078faf7692975fb8c8770f5004b WatchSource:0}: Error finding container 4a8cb0e56d4e0d6a4bd7bff4a9f43d79e4355078faf7692975fb8c8770f5004b: Status 404 returned error can't find the container with id 4a8cb0e56d4e0d6a4bd7bff4a9f43d79e4355078faf7692975fb8c8770f5004b Oct 30 18:55:28.945369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.945339 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:55:28.945423 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:28.945399 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:55:29.141778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:29.141751 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="518d323a-10b1-4f21-966e-8454d6e37a52" path="/var/lib/kubelet/pods/518d323a-10b1-4f21-966e-8454d6e37a52/volumes" Oct 30 18:55:29.239280 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:29.239208 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c","Type":"ContainerStarted","Data":"f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84"} Oct 30 18:55:29.239280 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:29.239260 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c","Type":"ContainerStarted","Data":"4a8cb0e56d4e0d6a4bd7bff4a9f43d79e4355078faf7692975fb8c8770f5004b"} Oct 30 18:55:29.280788 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:29.280741 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.280728057 podStartE2EDuration="1.280728057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:29.28056904 +0000 UTC m=+3542.268909527" watchObservedRunningTime="2026-10-30 18:55:29.280728057 +0000 UTC m=+3542.269068541" Oct 30 18:55:32.698257 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:32.698220 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:32.726018 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:32.725958 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:32.799343 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:32.799319 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:32.799451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:32.799410 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:32.813154 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:32.813128 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:32.914998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:32.914965 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/26c702426f1aeab2e9e6116e994cbd362b1230f690bb6d32002e36ee94fa3aec/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:32.941332 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:32.941312 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:33.132657 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:33.132625 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-b7kx8\"" Oct 30 18:55:33.139684 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:33.139654 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:33.469585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:33.469512 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:55:33.472994 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:55:33.472967 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f61b21f_c3f8_456b_957e_0cdbb36a3e8f.slice/crio-41c73e5cee9424559c12b45b3aacf34fbc6e348ea5781f4f3d59c022e157585b WatchSource:0}: Error finding container 41c73e5cee9424559c12b45b3aacf34fbc6e348ea5781f4f3d59c022e157585b: Status 404 returned error can't find the container with id 41c73e5cee9424559c12b45b3aacf34fbc6e348ea5781f4f3d59c022e157585b Oct 30 18:55:34.271538 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:34.271477 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f","Type":"ContainerStarted","Data":"1692c5192d2f8513a29e9261f35592b07e7ad1f39e2490be08c6f7205c9022ca"} Oct 30 18:55:34.271538 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:34.271535 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f","Type":"ContainerStarted","Data":"41c73e5cee9424559c12b45b3aacf34fbc6e348ea5781f4f3d59c022e157585b"} Oct 30 18:55:38.293412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:38.293382 5014 generic.go:356] "Generic (PLEG): container finished" podID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerID="1692c5192d2f8513a29e9261f35592b07e7ad1f39e2490be08c6f7205c9022ca" exitCode=0 Oct 30 18:55:38.293797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:38.293453 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f","Type":"ContainerDied","Data":"1692c5192d2f8513a29e9261f35592b07e7ad1f39e2490be08c6f7205c9022ca"} Oct 30 18:55:39.300576 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:39.300542 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f","Type":"ContainerStarted","Data":"4d9f0fa18811667baa3b2b8503706c4734fb1ea7049815dab5dbd1d1cf0136c1"} Oct 30 18:55:39.301064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:39.300646 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f","Type":"ContainerStarted","Data":"7505df59ca5b508453dc3332b0c0c5374632c453fee94b7de7657d99884ae84c"} Oct 30 18:55:39.301064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:39.300689 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f","Type":"ContainerStarted","Data":"cfd9cd10643fd6aa3a7dcd4df0f7a3cc3df1635830e50e672693c308f70a0359"} Oct 30 18:55:39.379560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:39.379516 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=11.494536708 podStartE2EDuration="15.379503314s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.884966606s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:39.366883532 +0000 UTC m=+3552.355224020" watchObservedRunningTime="2026-10-30 18:55:39.379503314 +0000 UTC m=+3552.367843801" Oct 30 18:55:43.142150 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:43.142116 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:43.334057 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:55:43.334027 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:55:48.140238 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:48.140209 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:48.145613 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:48.145573 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:55:48.345239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:55:48.345217 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:04.373177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:04.373136 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:04.373828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:04.373364 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" containerName="otc-container" containerID="cri-o://f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84" gracePeriod=30 Oct 30 18:56:06.727249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.727225 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:06.778072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.778046 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-data-science-collector-ta-client-cert\") pod \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " Oct 30 18:56:06.778207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.778124 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-monitoring-tls-certs\") pod \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " Oct 30 18:56:06.778207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.778157 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otc-internal\") pod \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " Oct 30 18:56:06.778207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.778183 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-tls-certs\") pod \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " Oct 30 18:56:06.778441 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.778230 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-kube-api-access-2dbgp\" (UniqueName: \"kubernetes.io/projected/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-kube-api-access-2dbgp\") pod \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " Oct 30 18:56:06.778441 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.778317 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otlp-tls-certs\") pod \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\" (UID: \"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c\") " Oct 30 18:56:06.778576 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.778550 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otc-internal" pod "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" (UID: "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:06.781279 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.781251 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-monitoring-tls-certs" pod "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" (UID: "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:06.781378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.781351 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-data-science-collector-ta-client-cert" pod "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" (UID: "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:06.781649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.781627 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-kube-api-access-2dbgp" pod "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" (UID: "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c"). InnerVolumeSpecName "kube-api-access-2dbgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:06.782441 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.782417 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-tls-certs" pod "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" (UID: "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:06.782550 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.782524 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otlp-tls-certs" pod "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" (UID: "64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:06.879277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.879251 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:06.879277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.879275 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dbgp\" (UniqueName: \"kubernetes.io/projected/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-kube-api-access-2dbgp\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:06.879434 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.879286 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:06.879434 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.879297 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:06.879434 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.879309 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:06.879434 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:06.879317 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:07.434740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.434672 5014 generic.go:356] "Generic (PLEG): container finished" podID="64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" containerID="f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84" exitCode=0 Oct 30 18:56:07.434740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.434729 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.434912 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.434754 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c","Type":"ContainerDied","Data":"f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84"} Oct 30 18:56:07.434912 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.434807 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c","Type":"ContainerDied","Data":"4a8cb0e56d4e0d6a4bd7bff4a9f43d79e4355078faf7692975fb8c8770f5004b"} Oct 30 18:56:07.434912 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.434829 5014 scope.go:122] "RemoveContainer" containerID="f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84" Oct 30 18:56:07.443797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.443770 5014 scope.go:122] "RemoveContainer" containerID="f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84" Oct 30 18:56:07.444058 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:56:07.444039 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84\": container with ID starting with f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84 not found: ID does not exist" containerID="f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84" Oct 30 18:56:07.444134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.444065 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84"} err="failed to get container status \"f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84\": rpc error: code = NotFound desc = could not find container \"f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84\": container with ID starting with f2719b98c6df4225b86eac885e7661d5257ac2e239efa38131f55865a08c6d84 not found: ID does not exist" Oct 30 18:56:07.488239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.488220 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:07.499635 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.499614 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:07.531138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.531115 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:07.531566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.531550 5014 cpu_manager.go:414] "containerMap: removing container" podUID="64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" containerName="otc-container" Oct 30 18:56:07.531655 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.531569 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" containerName="otc-container" Oct 30 18:56:07.531713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.531701 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" containerName="otc-container" Oct 30 18:56:07.536089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.536071 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.550530 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.550504 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:56:07.550648 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.550532 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:56:07.550648 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.550583 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:56:07.550648 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.550605 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cnw9r\"" Oct 30 18:56:07.550825 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.550738 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:56:07.550825 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.550724 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:56:07.564357 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.564328 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:07.585416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.585391 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knq5\" (UniqueName: \"kubernetes.io/projected/cf4898ea-2985-46f8-915a-cbe12a6f22d2-kube-api-access-2knq5\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.585537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.585425 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.585537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.585446 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.585537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.585494 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.585748 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.585552 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.585748 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.585639 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.686072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.686019 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knq5\" (UniqueName: \"kubernetes.io/projected/cf4898ea-2985-46f8-915a-cbe12a6f22d2-kube-api-access-2knq5\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.686072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.686053 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.686196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.686074 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.686196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.686098 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.686196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.686141 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.686196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.686183 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.686798 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.686770 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.688853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.688825 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.688943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.688855 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.688943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.688874 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.688943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.688899 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.709122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.709100 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knq5\" (UniqueName: \"kubernetes.io/projected/cf4898ea-2985-46f8-915a-cbe12a6f22d2-kube-api-access-2knq5\") pod \"data-science-collector-collector-0\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:07.848639 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:07.848616 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:08.144828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:08.144773 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:08.148654 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:56:08.148629 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4898ea_2985_46f8_915a_cbe12a6f22d2.slice/crio-098070794a661b1e7c30052775ab64367209eef5c9d45b9c75fe862518a8db3f WatchSource:0}: Error finding container 098070794a661b1e7c30052775ab64367209eef5c9d45b9c75fe862518a8db3f: Status 404 returned error can't find the container with id 098070794a661b1e7c30052775ab64367209eef5c9d45b9c75fe862518a8db3f Oct 30 18:56:08.150529 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:08.150501 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:56:08.150632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:08.150558 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:56:08.442476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:08.442386 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cf4898ea-2985-46f8-915a-cbe12a6f22d2","Type":"ContainerStarted","Data":"65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d"} Oct 30 18:56:08.442476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:08.442452 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cf4898ea-2985-46f8-915a-cbe12a6f22d2","Type":"ContainerStarted","Data":"098070794a661b1e7c30052775ab64367209eef5c9d45b9c75fe862518a8db3f"} Oct 30 18:56:08.485516 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:08.485463 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.485449629 podStartE2EDuration="1.485449629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:08.484567842 +0000 UTC m=+3581.472908333" watchObservedRunningTime="2026-10-30 18:56:08.485449629 +0000 UTC m=+3581.473790116" Oct 30 18:56:09.142169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.142135 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c" path="/var/lib/kubelet/pods/64bbb2f4-aeb6-48f5-bb67-9dab8ac2303c/volumes" Oct 30 18:56:09.287305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.287277 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr"] Oct 30 18:56:09.287770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.287718 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" podUID="414090c8-84f5-4508-aec9-32e2a4ec18e8" containerName="kube-rbac-proxy" containerID="cri-o://a4ca2ad736f644d8948bb8fef1c4f8dd72c37980b2c4bcb7e1b453fecc1385c1" gracePeriod=30 Oct 30 18:56:09.298393 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.298370 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w"] Oct 30 18:56:09.298579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.298556 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="kube-rbac-proxy" containerID="cri-o://6478f76e798279e0d65a96f9918e2487d4b94573dc7368fea271968aa02ce2c1" gracePeriod=30 Oct 30 18:56:09.298708 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.298581 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="prom-label-proxy" containerID="cri-o://a275ea43398788de44b6f3f13cd152cb32984ce89cfb5473062c9e676d7b0e18" gracePeriod=30 Oct 30 18:56:09.298708 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.298581 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="method-gate" containerID="cri-o://856f15d1dc4552340e63e083d84e13e26b08c912cd4072dc0dde95caf6f99291" gracePeriod=30 Oct 30 18:56:09.452580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.452543 5014 generic.go:356] "Generic (PLEG): container finished" podID="414090c8-84f5-4508-aec9-32e2a4ec18e8" containerID="a4ca2ad736f644d8948bb8fef1c4f8dd72c37980b2c4bcb7e1b453fecc1385c1" exitCode=0 Oct 30 18:56:09.452758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.452675 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" event={"ID":"414090c8-84f5-4508-aec9-32e2a4ec18e8","Type":"ContainerDied","Data":"a4ca2ad736f644d8948bb8fef1c4f8dd72c37980b2c4bcb7e1b453fecc1385c1"} Oct 30 18:56:09.456457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.456411 5014 generic.go:356] "Generic (PLEG): container finished" podID="89a16b76-3643-4050-8d72-5eaed55c463a" containerID="a275ea43398788de44b6f3f13cd152cb32984ce89cfb5473062c9e676d7b0e18" exitCode=0 Oct 30 18:56:09.456457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.456445 5014 generic.go:356] "Generic (PLEG): container finished" podID="89a16b76-3643-4050-8d72-5eaed55c463a" containerID="856f15d1dc4552340e63e083d84e13e26b08c912cd4072dc0dde95caf6f99291" exitCode=0 Oct 30 18:56:09.456457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.456455 5014 generic.go:356] "Generic (PLEG): container finished" podID="89a16b76-3643-4050-8d72-5eaed55c463a" containerID="6478f76e798279e0d65a96f9918e2487d4b94573dc7368fea271968aa02ce2c1" exitCode=0 Oct 30 18:56:09.456797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.456483 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" event={"ID":"89a16b76-3643-4050-8d72-5eaed55c463a","Type":"ContainerDied","Data":"a275ea43398788de44b6f3f13cd152cb32984ce89cfb5473062c9e676d7b0e18"} Oct 30 18:56:09.456797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.456556 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" event={"ID":"89a16b76-3643-4050-8d72-5eaed55c463a","Type":"ContainerDied","Data":"856f15d1dc4552340e63e083d84e13e26b08c912cd4072dc0dde95caf6f99291"} Oct 30 18:56:09.456797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.456610 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" event={"ID":"89a16b76-3643-4050-8d72-5eaed55c463a","Type":"ContainerDied","Data":"6478f76e798279e0d65a96f9918e2487d4b94573dc7368fea271968aa02ce2c1"} Oct 30 18:56:09.551307 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.551279 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:56:09.563051 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.563030 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:56:09.651265 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.651230 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l"] Oct 30 18:56:09.651520 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.651446 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" podUID="63258791-442a-4c75-9ecd-1cb65a8f0661" containerName="thanos-querier" containerID="cri-o://eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad" gracePeriod=30 Oct 30 18:56:09.704995 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.704967 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-prometheus-tls-ca\") pod \"89a16b76-3643-4050-8d72-5eaed55c463a\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " Oct 30 18:56:09.705149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705021 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-client-cert\") pod \"414090c8-84f5-4508-aec9-32e2a4ec18e8\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " Oct 30 18:56:09.705149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705043 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a16b76-3643-4050-8d72-5eaed55c463a-kube-api-access-xqfd7\" (UniqueName: \"kubernetes.io/projected/89a16b76-3643-4050-8d72-5eaed55c463a-kube-api-access-xqfd7\") pod \"89a16b76-3643-4050-8d72-5eaed55c463a\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " Oct 30 18:56:09.705297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705168 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-config\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-config\") pod \"89a16b76-3643-4050-8d72-5eaed55c463a\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " Oct 30 18:56:09.705297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705212 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-ca\") pod \"414090c8-84f5-4508-aec9-32e2a4ec18e8\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " Oct 30 18:56:09.705297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705255 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-rbac-proxy-config\") pod \"414090c8-84f5-4508-aec9-32e2a4ec18e8\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " Oct 30 18:56:09.705297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705289 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-api-access-sm9hl\" (UniqueName: \"kubernetes.io/projected/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-api-access-sm9hl\") pod \"414090c8-84f5-4508-aec9-32e2a4ec18e8\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " Oct 30 18:56:09.705585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705329 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89a16b76-3643-4050-8d72-5eaed55c463a-tls\" (UniqueName: \"kubernetes.io/secret/89a16b76-3643-4050-8d72-5eaed55c463a-tls\") pod \"89a16b76-3643-4050-8d72-5eaed55c463a\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " Oct 30 18:56:09.705585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705361 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-tls\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-tls\") pod \"414090c8-84f5-4508-aec9-32e2a4ec18e8\" (UID: \"414090c8-84f5-4508-aec9-32e2a4ec18e8\") " Oct 30 18:56:09.705585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705456 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-client-ca\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-client-ca\") pod \"89a16b76-3643-4050-8d72-5eaed55c463a\" (UID: \"89a16b76-3643-4050-8d72-5eaed55c463a\") " Oct 30 18:56:09.705585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705458 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-prometheus-tls-ca" pod "89a16b76-3643-4050-8d72-5eaed55c463a" (UID: "89a16b76-3643-4050-8d72-5eaed55c463a"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:09.705585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705519 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-config" pod "89a16b76-3643-4050-8d72-5eaed55c463a" (UID: "89a16b76-3643-4050-8d72-5eaed55c463a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:09.705585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705562 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-ca" pod "414090c8-84f5-4508-aec9-32e2a4ec18e8" (UID: "414090c8-84f5-4508-aec9-32e2a4ec18e8"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:09.706699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705942 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.706699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705963 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.706699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705977 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-prometheus-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.706699 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.705995 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-client-ca" pod "89a16b76-3643-4050-8d72-5eaed55c463a" (UID: "89a16b76-3643-4050-8d72-5eaed55c463a"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:09.707979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.707938 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a16b76-3643-4050-8d72-5eaed55c463a-kube-api-access-xqfd7" pod "89a16b76-3643-4050-8d72-5eaed55c463a" (UID: "89a16b76-3643-4050-8d72-5eaed55c463a"). InnerVolumeSpecName "kube-api-access-xqfd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:09.708095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.708077 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-client-cert" pod "414090c8-84f5-4508-aec9-32e2a4ec18e8" (UID: "414090c8-84f5-4508-aec9-32e2a4ec18e8"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:09.708293 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.708269 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-rbac-proxy-config" pod "414090c8-84f5-4508-aec9-32e2a4ec18e8" (UID: "414090c8-84f5-4508-aec9-32e2a4ec18e8"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:09.708293 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.708268 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a16b76-3643-4050-8d72-5eaed55c463a-tls" pod "89a16b76-3643-4050-8d72-5eaed55c463a" (UID: "89a16b76-3643-4050-8d72-5eaed55c463a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:09.708711 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.708693 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-tls" pod "414090c8-84f5-4508-aec9-32e2a4ec18e8" (UID: "414090c8-84f5-4508-aec9-32e2a4ec18e8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:09.709097 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.709080 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-api-access-sm9hl" pod "414090c8-84f5-4508-aec9-32e2a4ec18e8" (UID: "414090c8-84f5-4508-aec9-32e2a4ec18e8"). InnerVolumeSpecName "kube-api-access-sm9hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:09.777312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.777281 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:56:09.777808 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.777661 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="thanos-sidecar" containerID="cri-o://4d9f0fa18811667baa3b2b8503706c4734fb1ea7049815dab5dbd1d1cf0136c1" gracePeriod=600 Oct 30 18:56:09.777808 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.777693 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="config-reloader" containerID="cri-o://7505df59ca5b508453dc3332b0c0c5374632c453fee94b7de7657d99884ae84c" gracePeriod=600 Oct 30 18:56:09.777808 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.777626 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="prometheus" containerID="cri-o://cfd9cd10643fd6aa3a7dcd4df0f7a3cc3df1635830e50e672693c308f70a0359" gracePeriod=600 Oct 30 18:56:09.795128 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.795101 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:56:09.795406 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.795364 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="350afcca-4456-43a6-9087-3f8decb7942d" containerName="alertmanager" containerID="cri-o://cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86" gracePeriod=120 Oct 30 18:56:09.795566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.795550 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="350afcca-4456-43a6-9087-3f8decb7942d" containerName="config-reloader" containerID="cri-o://6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72" gracePeriod=120 Oct 30 18:56:09.806387 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.806362 5014 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/89a16b76-3643-4050-8d72-5eaed55c463a-client-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.806387 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.806385 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-prometheus-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.806552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.806397 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqfd7\" (UniqueName: \"kubernetes.io/projected/89a16b76-3643-4050-8d72-5eaed55c463a-kube-api-access-xqfd7\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.806552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.806406 5014 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-rbac-proxy-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.806552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.806415 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm9hl\" (UniqueName: \"kubernetes.io/projected/414090c8-84f5-4508-aec9-32e2a4ec18e8-kube-api-access-sm9hl\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.806552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.806424 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/89a16b76-3643-4050-8d72-5eaed55c463a-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.806552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.806432 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/414090c8-84f5-4508-aec9-32e2a4ec18e8-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.918659 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:09.912376 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" Oct 30 18:56:10.008348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.008318 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63258791-442a-4c75-9ecd-1cb65a8f0661-kube-api-access-xnlkj\" (UniqueName: \"kubernetes.io/projected/63258791-442a-4c75-9ecd-1cb65a8f0661-kube-api-access-xnlkj\") pod \"63258791-442a-4c75-9ecd-1cb65a8f0661\" (UID: \"63258791-442a-4c75-9ecd-1cb65a8f0661\") " Oct 30 18:56:10.010290 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.010264 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63258791-442a-4c75-9ecd-1cb65a8f0661-kube-api-access-xnlkj" pod "63258791-442a-4c75-9ecd-1cb65a8f0661" (UID: "63258791-442a-4c75-9ecd-1cb65a8f0661"). InnerVolumeSpecName "kube-api-access-xnlkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:10.109847 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.109823 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnlkj\" (UniqueName: \"kubernetes.io/projected/63258791-442a-4c75-9ecd-1cb65a8f0661-kube-api-access-xnlkj\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.405221 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.405132 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:10.465470 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.465433 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" Oct 30 18:56:10.465974 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.465761 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w" event={"ID":"89a16b76-3643-4050-8d72-5eaed55c463a","Type":"ContainerDied","Data":"ccba2728027096c3efb9a9f1b6c4ca4ba798ff78fe3601a66a883d1bdd6bb4fb"} Oct 30 18:56:10.466070 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.466018 5014 scope.go:122] "RemoveContainer" containerID="a275ea43398788de44b6f3f13cd152cb32984ce89cfb5473062c9e676d7b0e18" Oct 30 18:56:10.469769 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.469745 5014 generic.go:356] "Generic (PLEG): container finished" podID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerID="4d9f0fa18811667baa3b2b8503706c4734fb1ea7049815dab5dbd1d1cf0136c1" exitCode=0 Oct 30 18:56:10.469769 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.469770 5014 generic.go:356] "Generic (PLEG): container finished" podID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerID="7505df59ca5b508453dc3332b0c0c5374632c453fee94b7de7657d99884ae84c" exitCode=0 Oct 30 18:56:10.469903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.469784 5014 generic.go:356] "Generic (PLEG): container finished" podID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerID="cfd9cd10643fd6aa3a7dcd4df0f7a3cc3df1635830e50e672693c308f70a0359" exitCode=0 Oct 30 18:56:10.469903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.469878 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f","Type":"ContainerDied","Data":"4d9f0fa18811667baa3b2b8503706c4734fb1ea7049815dab5dbd1d1cf0136c1"} Oct 30 18:56:10.469979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.469936 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f","Type":"ContainerDied","Data":"7505df59ca5b508453dc3332b0c0c5374632c453fee94b7de7657d99884ae84c"} Oct 30 18:56:10.469979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.469974 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f","Type":"ContainerDied","Data":"cfd9cd10643fd6aa3a7dcd4df0f7a3cc3df1635830e50e672693c308f70a0359"} Oct 30 18:56:10.471305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.471280 5014 generic.go:356] "Generic (PLEG): container finished" podID="63258791-442a-4c75-9ecd-1cb65a8f0661" containerID="eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad" exitCode=0 Oct 30 18:56:10.471412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.471389 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" event={"ID":"63258791-442a-4c75-9ecd-1cb65a8f0661","Type":"ContainerDied","Data":"eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad"} Oct 30 18:56:10.471470 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.471415 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" event={"ID":"63258791-442a-4c75-9ecd-1cb65a8f0661","Type":"ContainerDied","Data":"9c03635c86bb6a61c22e04bce0a72bb1b59abc63861a4a3fd4f089ba96390661"} Oct 30 18:56:10.471528 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.471492 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l" Oct 30 18:56:10.481522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.481500 5014 scope.go:122] "RemoveContainer" containerID="856f15d1dc4552340e63e083d84e13e26b08c912cd4072dc0dde95caf6f99291" Oct 30 18:56:10.484280 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.484261 5014 generic.go:356] "Generic (PLEG): container finished" podID="350afcca-4456-43a6-9087-3f8decb7942d" containerID="6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72" exitCode=0 Oct 30 18:56:10.484581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.484326 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"350afcca-4456-43a6-9087-3f8decb7942d","Type":"ContainerDied","Data":"6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72"} Oct 30 18:56:10.486865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.486838 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="cf4898ea-2985-46f8-915a-cbe12a6f22d2" containerName="otc-container" containerID="cri-o://65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d" gracePeriod=30 Oct 30 18:56:10.487172 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.487153 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" Oct 30 18:56:10.489187 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.489156 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr" event={"ID":"414090c8-84f5-4508-aec9-32e2a4ec18e8","Type":"ContainerDied","Data":"924355ce619c1278a48cc6bde8f2e600a78272556c9e28cc786269d67bdf8c84"} Oct 30 18:56:10.503251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.503220 5014 scope.go:122] "RemoveContainer" containerID="6478f76e798279e0d65a96f9918e2487d4b94573dc7368fea271968aa02ce2c1" Oct 30 18:56:10.585240 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.585210 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w"] Oct 30 18:56:10.588956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.588930 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lz28w"] Oct 30 18:56:10.611086 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.610886 5014 scope.go:122] "RemoveContainer" containerID="eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad" Oct 30 18:56:10.618825 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.618803 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:10.622188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.622170 5014 scope.go:122] "RemoveContainer" containerID="eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad" Oct 30 18:56:10.622527 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:56:10.622497 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad\": container with ID starting with eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad not found: ID does not exist" containerID="eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad" Oct 30 18:56:10.622694 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.622541 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad"} err="failed to get container status \"eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad\": rpc error: code = NotFound desc = could not find container \"eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad\": container with ID starting with eb00f533d7c698f0d3c5d7f3ee8c0acfdb99e4b5038abaaa2f36d5bc4becb6ad not found: ID does not exist" Oct 30 18:56:10.622694 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.622564 5014 scope.go:122] "RemoveContainer" containerID="a4ca2ad736f644d8948bb8fef1c4f8dd72c37980b2c4bcb7e1b453fecc1385c1" Oct 30 18:56:10.641207 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.641170 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l"] Oct 30 18:56:10.646757 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.646735 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vb66l"] Oct 30 18:56:10.656682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.656633 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:10.686694 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.685696 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr"] Oct 30 18:56:10.689118 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.689093 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v28xr"] Oct 30 18:56:10.714882 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.714858 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.714893 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-secret-prometheus-web-tls-ca\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.714926 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.714957 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-kube-api-access-5bq5k\" (UniqueName: \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-kube-api-access-5bq5k\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.714982 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.715070 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.715131 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.715165 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.715195 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-thanos-prometheus-http-client-file\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.715255 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config-out\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.715252 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-1" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:10.715360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.715294 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.715360 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.715431 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-tls-assets\" (UniqueName: \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-tls-assets\") pod \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\" (UID: \"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f\") " Oct 30 18:56:10.715851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.715743 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.715851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.715767 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-0" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:10.716479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.716440 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-2" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:10.720733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.720618 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-secret-prometheus-web-tls-ca" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.720733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.720694 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-kube-api-access-5bq5k" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "kube-api-access-5bq5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:10.720988 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.720962 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.721480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.721456 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.721848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.721825 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-tls-assets" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:10.721995 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.721973 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "pvc-c02e8472-bb1a-4974-aa89-86eef715889b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:56:10.722163 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.722132 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-thanos-prometheus-http-client-file" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.722426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.722400 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.722524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.722444 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config-out" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:56:10.732770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.732748 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config" pod "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" (UID: "2f61b21f-c3f8-456b-957e-0cdbb36a3e8f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.782038 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.782014 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:10.816802 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.816776 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-alertmanager-data-science-monitoringstack-db\") pod \"350afcca-4456-43a6-9087-3f8decb7942d\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " Oct 30 18:56:10.816952 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.816812 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-tls-assets\" (UniqueName: \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-tls-assets\") pod \"350afcca-4456-43a6-9087-3f8decb7942d\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " Oct 30 18:56:10.816952 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.816857 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-config-volume\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-config-volume\") pod \"350afcca-4456-43a6-9087-3f8decb7942d\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " Oct 30 18:56:10.816952 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.816936 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-web-config\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-web-config\") pod \"350afcca-4456-43a6-9087-3f8decb7942d\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " Oct 30 18:56:10.817139 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.816965 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-kube-api-access-mnzjb\" (UniqueName: \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-kube-api-access-mnzjb\") pod \"350afcca-4456-43a6-9087-3f8decb7942d\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " Oct 30 18:56:10.817139 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.816987 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-config-out\") pod \"350afcca-4456-43a6-9087-3f8decb7942d\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " Oct 30 18:56:10.817139 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817040 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-cluster-tls-config\") pod \"350afcca-4456-43a6-9087-3f8decb7942d\" (UID: \"350afcca-4456-43a6-9087-3f8decb7942d\") " Oct 30 18:56:10.817285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817145 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-alertmanager-data-science-monitoringstack-db" pod "350afcca-4456-43a6-9087-3f8decb7942d" (UID: "350afcca-4456-43a6-9087-3f8decb7942d"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:56:10.817338 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817314 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817338 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817332 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817350 5014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817369 5014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817384 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817399 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817415 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817429 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817443 5014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817456 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817470 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bq5k\" (UniqueName: \"kubernetes.io/projected/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-kube-api-access-5bq5k\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817487 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.817869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.817515 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:56:10.819897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.819873 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-tls-assets" pod "350afcca-4456-43a6-9087-3f8decb7942d" (UID: "350afcca-4456-43a6-9087-3f8decb7942d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:10.820385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.820357 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-config-volume" pod "350afcca-4456-43a6-9087-3f8decb7942d" (UID: "350afcca-4456-43a6-9087-3f8decb7942d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.820622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.820570 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-config-out" pod "350afcca-4456-43a6-9087-3f8decb7942d" (UID: "350afcca-4456-43a6-9087-3f8decb7942d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:56:10.820622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.820576 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-kube-api-access-mnzjb" pod "350afcca-4456-43a6-9087-3f8decb7942d" (UID: "350afcca-4456-43a6-9087-3f8decb7942d"). InnerVolumeSpecName "kube-api-access-mnzjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:10.826441 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.826421 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-cluster-tls-config" pod "350afcca-4456-43a6-9087-3f8decb7942d" (UID: "350afcca-4456-43a6-9087-3f8decb7942d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.830695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.830677 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c02e8472-bb1a-4974-aa89-86eef715889b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:56:10.833753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.833622 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-web-config" pod "350afcca-4456-43a6-9087-3f8decb7942d" (UID: "350afcca-4456-43a6-9087-3f8decb7942d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.918386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918320 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otlp-tls-certs\") pod \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " Oct 30 18:56:10.918386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918365 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4898ea-2985-46f8-915a-cbe12a6f22d2-kube-api-access-2knq5\" (UniqueName: \"kubernetes.io/projected/cf4898ea-2985-46f8-915a-cbe12a6f22d2-kube-api-access-2knq5\") pod \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " Oct 30 18:56:10.918614 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918411 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-monitoring-tls-certs\") pod \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " Oct 30 18:56:10.918614 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918444 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-data-science-collector-ta-client-cert\") pod \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " Oct 30 18:56:10.918614 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918465 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otc-internal\") pod \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " Oct 30 18:56:10.918614 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918489 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-tls-certs\") pod \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\" (UID: \"cf4898ea-2985-46f8-915a-cbe12a6f22d2\") " Oct 30 18:56:10.918852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918697 5014 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-config-volume\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.918852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918716 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.918852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918731 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzjb\" (UniqueName: \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-kube-api-access-mnzjb\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.918852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918745 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/350afcca-4456-43a6-9087-3f8decb7942d-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.918852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918758 5014 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/350afcca-4456-43a6-9087-3f8decb7942d-cluster-tls-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.918852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918775 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" Oct 30 18:56:10.918852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918789 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/350afcca-4456-43a6-9087-3f8decb7942d-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.919232 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.918873 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otc-internal" pod "cf4898ea-2985-46f8-915a-cbe12a6f22d2" (UID: "cf4898ea-2985-46f8-915a-cbe12a6f22d2"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:10.920689 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.920662 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otlp-tls-certs" pod "cf4898ea-2985-46f8-915a-cbe12a6f22d2" (UID: "cf4898ea-2985-46f8-915a-cbe12a6f22d2"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.920689 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.920679 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-data-science-collector-ta-client-cert" pod "cf4898ea-2985-46f8-915a-cbe12a6f22d2" (UID: "cf4898ea-2985-46f8-915a-cbe12a6f22d2"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.920847 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.920717 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4898ea-2985-46f8-915a-cbe12a6f22d2-kube-api-access-2knq5" pod "cf4898ea-2985-46f8-915a-cbe12a6f22d2" (UID: "cf4898ea-2985-46f8-915a-cbe12a6f22d2"). InnerVolumeSpecName "kube-api-access-2knq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:10.921039 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.921023 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-monitoring-tls-certs" pod "cf4898ea-2985-46f8-915a-cbe12a6f22d2" (UID: "cf4898ea-2985-46f8-915a-cbe12a6f22d2"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.921097 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:10.921045 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-tls-certs" pod "cf4898ea-2985-46f8-915a-cbe12a6f22d2" (UID: "cf4898ea-2985-46f8-915a-cbe12a6f22d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:11.019307 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.019282 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:11.019307 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.019304 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2knq5\" (UniqueName: \"kubernetes.io/projected/cf4898ea-2985-46f8-915a-cbe12a6f22d2-kube-api-access-2knq5\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:11.019450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.019313 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:11.019450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.019323 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:11.019450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.019334 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cf4898ea-2985-46f8-915a-cbe12a6f22d2-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:11.019450 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.019343 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4898ea-2985-46f8-915a-cbe12a6f22d2-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:11.141249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.141221 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414090c8-84f5-4508-aec9-32e2a4ec18e8" path="/var/lib/kubelet/pods/414090c8-84f5-4508-aec9-32e2a4ec18e8/volumes" Oct 30 18:56:11.141521 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.141509 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63258791-442a-4c75-9ecd-1cb65a8f0661" path="/var/lib/kubelet/pods/63258791-442a-4c75-9ecd-1cb65a8f0661/volumes" Oct 30 18:56:11.141881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.141867 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" path="/var/lib/kubelet/pods/89a16b76-3643-4050-8d72-5eaed55c463a/volumes" Oct 30 18:56:11.494392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.494355 5014 generic.go:356] "Generic (PLEG): container finished" podID="350afcca-4456-43a6-9087-3f8decb7942d" containerID="cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86" exitCode=0 Oct 30 18:56:11.494826 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.494423 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:11.494826 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.494436 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"350afcca-4456-43a6-9087-3f8decb7942d","Type":"ContainerDied","Data":"cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86"} Oct 30 18:56:11.494826 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.494514 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"350afcca-4456-43a6-9087-3f8decb7942d","Type":"ContainerDied","Data":"8b05690a577e499db65b28afb52799d5d42d344d2ab7bd4ce3c4cca126786379"} Oct 30 18:56:11.494826 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.494546 5014 scope.go:122] "RemoveContainer" containerID="6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72" Oct 30 18:56:11.499088 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.499056 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2f61b21f-c3f8-456b-957e-0cdbb36a3e8f","Type":"ContainerDied","Data":"41c73e5cee9424559c12b45b3aacf34fbc6e348ea5781f4f3d59c022e157585b"} Oct 30 18:56:11.499214 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.499062 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:11.501578 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.501554 5014 generic.go:356] "Generic (PLEG): container finished" podID="cf4898ea-2985-46f8-915a-cbe12a6f22d2" containerID="65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d" exitCode=0 Oct 30 18:56:11.501733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.501616 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cf4898ea-2985-46f8-915a-cbe12a6f22d2","Type":"ContainerDied","Data":"65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d"} Oct 30 18:56:11.501733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.501651 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cf4898ea-2985-46f8-915a-cbe12a6f22d2","Type":"ContainerDied","Data":"098070794a661b1e7c30052775ab64367209eef5c9d45b9c75fe862518a8db3f"} Oct 30 18:56:11.501733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.501697 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:11.504885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.504866 5014 scope.go:122] "RemoveContainer" containerID="cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86" Oct 30 18:56:11.513639 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.513624 5014 scope.go:122] "RemoveContainer" containerID="5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9" Oct 30 18:56:11.521617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.521579 5014 scope.go:122] "RemoveContainer" containerID="6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72" Oct 30 18:56:11.521893 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:56:11.521875 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72\": container with ID starting with 6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72 not found: ID does not exist" containerID="6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72" Oct 30 18:56:11.521973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.521903 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72"} err="failed to get container status \"6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72\": rpc error: code = NotFound desc = could not find container \"6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72\": container with ID starting with 6830afc682fb462fe6ccf8517a0dbe9c663fd736b4ef04a1f8eba6e9efd77b72 not found: ID does not exist" Oct 30 18:56:11.521973 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.521926 5014 scope.go:122] "RemoveContainer" containerID="cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86" Oct 30 18:56:11.522156 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:56:11.522140 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86\": container with ID starting with cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86 not found: ID does not exist" containerID="cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86" Oct 30 18:56:11.522223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.522164 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86"} err="failed to get container status \"cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86\": rpc error: code = NotFound desc = could not find container \"cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86\": container with ID starting with cb10ab0cf0bb4078b14d2a63f581d81c73b9b5c4f4360ea84b661c4792212c86 not found: ID does not exist" Oct 30 18:56:11.522223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.522183 5014 scope.go:122] "RemoveContainer" containerID="5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9" Oct 30 18:56:11.522424 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:56:11.522405 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9\": container with ID starting with 5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9 not found: ID does not exist" containerID="5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9" Oct 30 18:56:11.522463 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.522431 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9"} err="failed to get container status \"5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9\": rpc error: code = NotFound desc = could not find container \"5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9\": container with ID starting with 5309a01aacbd7fd018770d94785725a0ce407d5c8a7a1be065eb2b563a0b5de9 not found: ID does not exist" Oct 30 18:56:11.522463 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.522446 5014 scope.go:122] "RemoveContainer" containerID="4d9f0fa18811667baa3b2b8503706c4734fb1ea7049815dab5dbd1d1cf0136c1" Oct 30 18:56:11.529885 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.529865 5014 scope.go:122] "RemoveContainer" containerID="7505df59ca5b508453dc3332b0c0c5374632c453fee94b7de7657d99884ae84c" Oct 30 18:56:11.537682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.537666 5014 scope.go:122] "RemoveContainer" containerID="cfd9cd10643fd6aa3a7dcd4df0f7a3cc3df1635830e50e672693c308f70a0359" Oct 30 18:56:11.545725 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.545709 5014 scope.go:122] "RemoveContainer" containerID="1692c5192d2f8513a29e9261f35592b07e7ad1f39e2490be08c6f7205c9022ca" Oct 30 18:56:11.547259 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.547239 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:56:11.551433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.551411 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:56:11.556242 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.556227 5014 scope.go:122] "RemoveContainer" containerID="65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d" Oct 30 18:56:11.563771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.563754 5014 scope.go:122] "RemoveContainer" containerID="65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d" Oct 30 18:56:11.564028 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:56:11.564001 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d\": container with ID starting with 65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d not found: ID does not exist" containerID="65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d" Oct 30 18:56:11.564107 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.564038 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d"} err="failed to get container status \"65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d\": rpc error: code = NotFound desc = could not find container \"65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d\": container with ID starting with 65a0ae18039ed9f778826b1408c675d2b9296f4546f32584a1dee168626b307d not found: ID does not exist" Oct 30 18:56:11.584969 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.584946 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:11.592462 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.592439 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:11.626171 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.626151 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:56:11.632646 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:11.632626 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:56:13.143457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:13.143415 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" path="/var/lib/kubelet/pods/2f61b21f-c3f8-456b-957e-0cdbb36a3e8f/volumes" Oct 30 18:56:13.143982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:13.143963 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="350afcca-4456-43a6-9087-3f8decb7942d" path="/var/lib/kubelet/pods/350afcca-4456-43a6-9087-3f8decb7942d/volumes" Oct 30 18:56:13.144792 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:13.144771 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4898ea-2985-46f8-915a-cbe12a6f22d2" path="/var/lib/kubelet/pods/cf4898ea-2985-46f8-915a-cbe12a6f22d2/volumes" Oct 30 18:56:13.334143 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:56:13.334107 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:56:20.539340 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539307 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx"] Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539627 5014 cpu_manager.go:414] "containerMap: removing container" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="method-gate" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539639 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="method-gate" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539647 5014 cpu_manager.go:414] "containerMap: removing container" podUID="350afcca-4456-43a6-9087-3f8decb7942d" containerName="alertmanager" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539655 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350afcca-4456-43a6-9087-3f8decb7942d" containerName="alertmanager" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539664 5014 cpu_manager.go:414] "containerMap: removing container" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="config-reloader" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539671 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="config-reloader" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539687 5014 cpu_manager.go:414] "containerMap: removing container" podUID="414090c8-84f5-4508-aec9-32e2a4ec18e8" containerName="kube-rbac-proxy" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539694 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414090c8-84f5-4508-aec9-32e2a4ec18e8" containerName="kube-rbac-proxy" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539711 5014 cpu_manager.go:414] "containerMap: removing container" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="prometheus" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539717 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="prometheus" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539733 5014 cpu_manager.go:414] "containerMap: removing container" podUID="350afcca-4456-43a6-9087-3f8decb7942d" containerName="init-config-reloader" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539740 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350afcca-4456-43a6-9087-3f8decb7942d" containerName="init-config-reloader" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539749 5014 cpu_manager.go:414] "containerMap: removing container" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="init-config-reloader" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539756 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="init-config-reloader" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539770 5014 cpu_manager.go:414] "containerMap: removing container" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="prom-label-proxy" Oct 30 18:56:20.539773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539777 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="prom-label-proxy" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539792 5014 cpu_manager.go:414] "containerMap: removing container" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="kube-rbac-proxy" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539799 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="kube-rbac-proxy" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539809 5014 cpu_manager.go:414] "containerMap: removing container" podUID="350afcca-4456-43a6-9087-3f8decb7942d" containerName="config-reloader" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539815 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350afcca-4456-43a6-9087-3f8decb7942d" containerName="config-reloader" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539827 5014 cpu_manager.go:414] "containerMap: removing container" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="thanos-sidecar" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539833 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="thanos-sidecar" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539845 5014 cpu_manager.go:414] "containerMap: removing container" podUID="63258791-442a-4c75-9ecd-1cb65a8f0661" containerName="thanos-querier" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539852 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63258791-442a-4c75-9ecd-1cb65a8f0661" containerName="thanos-querier" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539868 5014 cpu_manager.go:414] "containerMap: removing container" podUID="cf4898ea-2985-46f8-915a-cbe12a6f22d2" containerName="otc-container" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539876 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4898ea-2985-46f8-915a-cbe12a6f22d2" containerName="otc-container" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539972 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="prom-label-proxy" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539988 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="kube-rbac-proxy" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.539998 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="350afcca-4456-43a6-9087-3f8decb7942d" containerName="config-reloader" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.540008 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="thanos-sidecar" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.540016 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="63258791-442a-4c75-9ecd-1cb65a8f0661" containerName="thanos-querier" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.540024 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4898ea-2985-46f8-915a-cbe12a6f22d2" containerName="otc-container" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.540034 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a16b76-3643-4050-8d72-5eaed55c463a" containerName="method-gate" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.540042 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="350afcca-4456-43a6-9087-3f8decb7942d" containerName="alertmanager" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.540047 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="config-reloader" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.540053 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="414090c8-84f5-4508-aec9-32e2a4ec18e8" containerName="kube-rbac-proxy" Oct 30 18:56:20.540380 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.540058 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f61b21f-c3f8-456b-957e-0cdbb36a3e8f" containerName="prometheus" Oct 30 18:56:20.544824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.544800 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.559057 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.559034 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:56:20.570389 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.570370 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:56:20.570564 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.570540 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 18:56:20.570682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.570617 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 18:56:20.570731 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.570692 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-cgh8p\"" Oct 30 18:56:20.571812 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.571794 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx"] Oct 30 18:56:20.577954 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.577937 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw"] Oct 30 18:56:20.581505 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.581490 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.605708 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.605413 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw"] Oct 30 18:56:20.659413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.659389 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 18:56:20.659413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.659407 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:56:20.659585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.659429 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:56:20.659585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.659491 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 18:56:20.662490 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.662473 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 18:56:20.662728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.662710 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-5qcsn\"" Oct 30 18:56:20.689123 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.689101 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4b5\" (UniqueName: \"kubernetes.io/projected/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-kube-api-access-7x4b5\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.689216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.689139 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.689216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.689167 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.689343 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.689323 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.689388 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.689365 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4htjd\" (UniqueName: \"kubernetes.io/projected/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-api-access-4htjd\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.689428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.689398 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.689469 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.689436 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.689469 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.689460 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.689545 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.689519 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.689585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.689541 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.790384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.790325 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.790384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.790362 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.790384 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.790383 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.790644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.790408 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4htjd\" (UniqueName: \"kubernetes.io/projected/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-api-access-4htjd\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.790644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.790434 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.790644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.790460 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.790644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.790491 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.790644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.790515 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.790644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.790544 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.790644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.790576 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4b5\" (UniqueName: \"kubernetes.io/projected/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-kube-api-access-7x4b5\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.791045 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:56:20.790844 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 18:56:20.791045 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:56:20.790916 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-client-cert podName:63ea83bc-7552-4cc1-805f-9f5eb6d10513 nodeName:}" failed. No retries permitted until 2026-10-30 18:56:21.290897954 +0000 UTC m=+3594.279238438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-65lcx" (UID: "63ea83bc-7552-4cc1-805f-9f5eb6d10513") : secret "prometheus-operated-tls" not found Oct 30 18:56:20.791186 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.791161 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.791300 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.791171 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.791417 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.791245 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.791891 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.791869 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.793138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.793108 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.793211 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.793139 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.793249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.793215 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.815409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.815387 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4b5\" (UniqueName: \"kubernetes.io/projected/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-kube-api-access-7x4b5\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:20.815551 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.815534 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4htjd\" (UniqueName: \"kubernetes.io/projected/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-api-access-4htjd\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:20.899872 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:20.899854 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:56:21.207753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:21.207726 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw"] Oct 30 18:56:21.212430 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:56:21.212406 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2be25a_2bc2_4b8a_bfe1_0be5ed20428e.slice/crio-30c1c129ee366b62d01fc14ad5486038cd3d2c658862835db10b21528d0442b3 WatchSource:0}: Error finding container 30c1c129ee366b62d01fc14ad5486038cd3d2c658862835db10b21528d0442b3: Status 404 returned error can't find the container with id 30c1c129ee366b62d01fc14ad5486038cd3d2c658862835db10b21528d0442b3 Oct 30 18:56:21.294374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:21.294348 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:21.296556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:21.296536 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-65lcx\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:21.457805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:21.457748 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:21.552897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:21.552866 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" event={"ID":"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e","Type":"ContainerStarted","Data":"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d"} Oct 30 18:56:21.553237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:21.552929 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" event={"ID":"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e","Type":"ContainerStarted","Data":"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44"} Oct 30 18:56:21.553237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:21.552964 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" event={"ID":"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e","Type":"ContainerStarted","Data":"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5"} Oct 30 18:56:21.553237 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:21.552989 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" event={"ID":"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e","Type":"ContainerStarted","Data":"30c1c129ee366b62d01fc14ad5486038cd3d2c658862835db10b21528d0442b3"} Oct 30 18:56:21.654286 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:21.654235 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" podStartSLOduration=1.654221921 podStartE2EDuration="1.654221921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:21.652771239 +0000 UTC m=+3594.641111726" watchObservedRunningTime="2026-10-30 18:56:21.654221921 +0000 UTC m=+3594.642562408" Oct 30 18:56:21.755504 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:21.755429 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx"] Oct 30 18:56:21.760528 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:56:21.760491 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ea83bc_7552_4cc1_805f_9f5eb6d10513.slice/crio-578374ba907bc82b90bf280452133d5f85a26cbd31f558d6dfd4a0b91941d0f3 WatchSource:0}: Error finding container 578374ba907bc82b90bf280452133d5f85a26cbd31f558d6dfd4a0b91941d0f3: Status 404 returned error can't find the container with id 578374ba907bc82b90bf280452133d5f85a26cbd31f558d6dfd4a0b91941d0f3 Oct 30 18:56:22.086844 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.086818 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:22.091559 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.091535 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.106134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.106102 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:56:22.117072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.117049 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:56:22.117194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.117075 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:56:22.117194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.117133 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:56:22.117194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.117142 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-p8bgw\"" Oct 30 18:56:22.117504 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.117490 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:56:22.119026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.119010 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:22.204148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.204116 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmjpb\" (UniqueName: \"kubernetes.io/projected/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-kube-api-access-hmjpb\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.204316 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.204176 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.204394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.204325 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.204394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.204378 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.204537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.204433 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.204537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.204473 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.305485 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.305443 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.305485 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.305488 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.305762 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.305529 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.305762 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.305563 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.305762 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.305634 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmjpb\" (UniqueName: \"kubernetes.io/projected/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-kube-api-access-hmjpb\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.305762 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.305670 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.306360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.306333 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.308185 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.308162 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.308316 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.308295 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.308568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.308552 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.308644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.308626 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.328681 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.328654 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmjpb\" (UniqueName: \"kubernetes.io/projected/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-kube-api-access-hmjpb\") pod \"data-science-collector-collector-0\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.409278 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.409260 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:22.559203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.559165 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" event={"ID":"63ea83bc-7552-4cc1-805f-9f5eb6d10513","Type":"ContainerStarted","Data":"677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5"} Oct 30 18:56:22.559655 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.559221 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" event={"ID":"63ea83bc-7552-4cc1-805f-9f5eb6d10513","Type":"ContainerStarted","Data":"578374ba907bc82b90bf280452133d5f85a26cbd31f558d6dfd4a0b91941d0f3"} Oct 30 18:56:22.559655 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.559377 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:22.599707 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.599661 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" podStartSLOduration=2.599644951 podStartE2EDuration="2.599644951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:22.598914617 +0000 UTC m=+3595.587255105" watchObservedRunningTime="2026-10-30 18:56:22.599644951 +0000 UTC m=+3595.587985439" Oct 30 18:56:22.708096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.708036 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:22.712364 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:56:22.712322 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4031a1c8_25ce_46fd_8f39_99d17c9d2f43.slice/crio-d30ea0e17bacca3697dd4f27a541d36bd8febee53bd5d258faa8681d8cb44b17 WatchSource:0}: Error finding container d30ea0e17bacca3697dd4f27a541d36bd8febee53bd5d258faa8681d8cb44b17: Status 404 returned error can't find the container with id d30ea0e17bacca3697dd4f27a541d36bd8febee53bd5d258faa8681d8cb44b17 Oct 30 18:56:22.714693 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.714667 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:56:22.714785 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:22.714720 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:56:23.566392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:23.566352 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4031a1c8-25ce-46fd-8f39-99d17c9d2f43","Type":"ContainerStarted","Data":"61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb"} Oct 30 18:56:23.566860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:23.566414 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4031a1c8-25ce-46fd-8f39-99d17c9d2f43","Type":"ContainerStarted","Data":"d30ea0e17bacca3697dd4f27a541d36bd8febee53bd5d258faa8681d8cb44b17"} Oct 30 18:56:23.610440 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:23.610390 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.610375199 podStartE2EDuration="1.610375199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:23.609557717 +0000 UTC m=+3596.597898205" watchObservedRunningTime="2026-10-30 18:56:23.610375199 +0000 UTC m=+3596.598715686" Oct 30 18:56:27.479658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:27.479536 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:56:27.526022 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:27.490362 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 18:56:28.569058 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:28.569027 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:56:29.190219 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:29.190183 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8"] Oct 30 18:56:29.195467 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:29.195446 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" Oct 30 18:56:29.212141 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:29.212121 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-5mtd6\"" Oct 30 18:56:29.213502 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:29.213484 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8"] Oct 30 18:56:29.358165 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:29.358137 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlbl\" (UniqueName: \"kubernetes.io/projected/f2e66b4d-3b28-4974-84e8-c1a31773f2c4-kube-api-access-2rlbl\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8\" (UID: \"f2e66b4d-3b28-4974-84e8-c1a31773f2c4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" Oct 30 18:56:29.459462 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:29.459404 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlbl\" (UniqueName: \"kubernetes.io/projected/f2e66b4d-3b28-4974-84e8-c1a31773f2c4-kube-api-access-2rlbl\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8\" (UID: \"f2e66b4d-3b28-4974-84e8-c1a31773f2c4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" Oct 30 18:56:29.482403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:29.482379 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlbl\" (UniqueName: \"kubernetes.io/projected/f2e66b4d-3b28-4974-84e8-c1a31773f2c4-kube-api-access-2rlbl\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8\" (UID: \"f2e66b4d-3b28-4974-84e8-c1a31773f2c4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" Oct 30 18:56:29.520710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:29.520689 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" Oct 30 18:56:29.819495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:29.819465 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8"] Oct 30 18:56:29.824066 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:56:29.824035 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2e66b4d_3b28_4974_84e8_c1a31773f2c4.slice/crio-4b39790664e5cb66e006ade991731eea631b9441902efe1635edbd3334f1f1df WatchSource:0}: Error finding container 4b39790664e5cb66e006ade991731eea631b9441902efe1635edbd3334f1f1df: Status 404 returned error can't find the container with id 4b39790664e5cb66e006ade991731eea631b9441902efe1635edbd3334f1f1df Oct 30 18:56:30.059272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.059227 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:56:30.064109 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.064064 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.095073 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.095043 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:56:30.152173 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.152124 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:56:30.152173 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.152149 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:56:30.152455 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.152194 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:56:30.152455 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.152297 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:56:30.152455 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.152391 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-7rf6z\"" Oct 30 18:56:30.152655 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.152455 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:56:30.156625 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.156584 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.165424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.165400 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.165524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.165466 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.165524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.165501 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.165665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.165643 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvxk\" (UniqueName: \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-kube-api-access-ccvxk\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.165707 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.165682 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.165753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.165739 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.165864 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.165849 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.185745 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.185705 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:56:30.234465 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.234445 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:56:30.234585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.234452 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:56:30.234585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.234479 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:56:30.253113 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.253091 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:56:30.253179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.253110 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:56:30.253179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.253146 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:56:30.253179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.253161 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:56:30.253302 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.253207 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-786bx\"" Oct 30 18:56:30.253302 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.253258 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:56:30.266541 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.266520 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.266633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.266555 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.266633 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.266579 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/116e8cd8-5f76-4b30-a446-0051cac4a571-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.266780 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.266645 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.266780 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.266671 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.266780 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.266698 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.266884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.266784 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.266884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.266852 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.266964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.266928 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.267013 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.266977 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.267102 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.267085 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.267176 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.267108 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvxk\" (UniqueName: \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-kube-api-access-ccvxk\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.267176 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.267146 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.267287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.267173 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.267287 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.267237 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.267399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.267276 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.267650 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.267412 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhprn\" (UniqueName: \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-kube-api-access-qhprn\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.267650 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.267537 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.267650 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.267587 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.267829 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.267659 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.269116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.269097 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.269220 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.269201 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.269575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.269553 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.269733 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.269718 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.270057 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.270041 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.289170 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.289153 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvxk\" (UniqueName: \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-kube-api-access-ccvxk\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.368924 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.368868 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.368924 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.368913 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/116e8cd8-5f76-4b30-a446-0051cac4a571-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.369067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.368938 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.369067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.368962 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.369067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.368992 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.369067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.369021 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.369067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.369062 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.369297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.369094 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.369297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.369120 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.369297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.369167 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.369297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.369197 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhprn\" (UniqueName: \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-kube-api-access-qhprn\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.369297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.369231 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.369748 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.369491 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.370069 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.370042 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.370555 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.370524 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.371710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.371684 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/116e8cd8-5f76-4b30-a446-0051cac4a571-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.372209 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.372185 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.372678 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.372658 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.372797 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.372775 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.372934 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.372917 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.373023 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.373008 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.373236 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.373216 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.373799 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.373783 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.383723 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.383702 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:30.392631 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.392586 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhprn\" (UniqueName: \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-kube-api-access-qhprn\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:30.603003 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.602966 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" event={"ID":"f2e66b4d-3b28-4974-84e8-c1a31773f2c4","Type":"ContainerStarted","Data":"1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22"} Oct 30 18:56:30.603153 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.603012 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" event={"ID":"f2e66b4d-3b28-4974-84e8-c1a31773f2c4","Type":"ContainerStarted","Data":"4b39790664e5cb66e006ade991731eea631b9441902efe1635edbd3334f1f1df"} Oct 30 18:56:30.641347 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.641270 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" podStartSLOduration=1.641257349 podStartE2EDuration="1.641257349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:30.640462788 +0000 UTC m=+3603.628803276" watchObservedRunningTime="2026-10-30 18:56:30.641257349 +0000 UTC m=+3603.629597835" Oct 30 18:56:30.695569 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:30.695536 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:56:30.701525 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:56:30.701500 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf1802_ffc1_4649_82a8_5a71b2827a01.slice/crio-9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec WatchSource:0}: Error finding container 9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec: Status 404 returned error can't find the container with id 9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec Oct 30 18:56:31.609144 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:31.609102 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"badf1802-ffc1-4649-82a8-5a71b2827a01","Type":"ContainerStarted","Data":"f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d"} Oct 30 18:56:31.609537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:31.609170 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"badf1802-ffc1-4649-82a8-5a71b2827a01","Type":"ContainerStarted","Data":"9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec"} Oct 30 18:56:33.697274 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:33.697242 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:33.724819 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:33.724792 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:33.798332 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:33.798302 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:33.798474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:33.798414 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:33.812262 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:33.812239 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:33.913015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:33.912989 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/26c702426f1aeab2e9e6116e994cbd362b1230f690bb6d32002e36ee94fa3aec/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:33.939538 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:33.939517 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:34.076978 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:34.076958 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:34.413499 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:56:34.413474 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116e8cd8_5f76_4b30_a446_0051cac4a571.slice/crio-88ca7dc837914e766f8bfe70f64f92b4c081d82b13856d8660dfbde13e984a7e WatchSource:0}: Error finding container 88ca7dc837914e766f8bfe70f64f92b4c081d82b13856d8660dfbde13e984a7e: Status 404 returned error can't find the container with id 88ca7dc837914e766f8bfe70f64f92b4c081d82b13856d8660dfbde13e984a7e Oct 30 18:56:34.421729 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:34.421710 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:56:34.623877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:34.623848 5014 generic.go:356] "Generic (PLEG): container finished" podID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerID="f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d" exitCode=0 Oct 30 18:56:34.624009 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:34.623921 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"badf1802-ffc1-4649-82a8-5a71b2827a01","Type":"ContainerDied","Data":"f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d"} Oct 30 18:56:34.625188 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:34.625164 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"116e8cd8-5f76-4b30-a446-0051cac4a571","Type":"ContainerStarted","Data":"88ca7dc837914e766f8bfe70f64f92b4c081d82b13856d8660dfbde13e984a7e"} Oct 30 18:56:35.632345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:35.632303 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"badf1802-ffc1-4649-82a8-5a71b2827a01","Type":"ContainerStarted","Data":"651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8"} Oct 30 18:56:35.632863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:35.632385 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"badf1802-ffc1-4649-82a8-5a71b2827a01","Type":"ContainerStarted","Data":"427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421"} Oct 30 18:56:35.632863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:35.632572 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:35.634079 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:35.634044 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"116e8cd8-5f76-4b30-a446-0051cac4a571","Type":"ContainerStarted","Data":"f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce"} Oct 30 18:56:35.704449 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:35.704408 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.418258317 podStartE2EDuration="5.704397201s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.286138884s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:35.692112812 +0000 UTC m=+3608.680453299" watchObservedRunningTime="2026-10-30 18:56:35.704397201 +0000 UTC m=+3608.692737688" Oct 30 18:56:39.653971 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:39.653935 5014 generic.go:356] "Generic (PLEG): container finished" podID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerID="f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce" exitCode=0 Oct 30 18:56:39.654351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:39.654006 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"116e8cd8-5f76-4b30-a446-0051cac4a571","Type":"ContainerDied","Data":"f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce"} Oct 30 18:56:40.661814 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:40.661779 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"116e8cd8-5f76-4b30-a446-0051cac4a571","Type":"ContainerStarted","Data":"3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513"} Oct 30 18:56:40.662192 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:40.661853 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"116e8cd8-5f76-4b30-a446-0051cac4a571","Type":"ContainerStarted","Data":"cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1"} Oct 30 18:56:40.662192 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:40.661880 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"116e8cd8-5f76-4b30-a446-0051cac4a571","Type":"ContainerStarted","Data":"d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b"} Oct 30 18:56:40.737670 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:40.737579 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=6.45291683 podStartE2EDuration="10.737567466s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.284650636s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:40.725460667 +0000 UTC m=+3613.713801166" watchObservedRunningTime="2026-10-30 18:56:40.737567466 +0000 UTC m=+3613.725907952" Oct 30 18:56:41.639553 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:41.639518 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:56:43.333254 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:56:43.333213 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:56:44.077445 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:44.077421 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:49.077853 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:49.077815 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:49.083860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:49.083834 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:49.708336 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:49.708300 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:56:55.711083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:55.711043 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:55.711566 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:55.711301 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="4031a1c8-25ce-46fd-8f39-99d17c9d2f43" containerName="otc-container" containerID="cri-o://61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb" gracePeriod=30 Oct 30 18:56:56.960091 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:56.960068 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:57.083534 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.083504 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-data-science-collector-ta-client-cert\") pod \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " Oct 30 18:56:57.083722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.083539 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otlp-tls-certs\") pod \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " Oct 30 18:56:57.083722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.083570 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-kube-api-access-hmjpb\" (UniqueName: \"kubernetes.io/projected/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-kube-api-access-hmjpb\") pod \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " Oct 30 18:56:57.083722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.083623 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-tls-certs\") pod \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " Oct 30 18:56:57.083722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.083660 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otc-internal\") pod \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " Oct 30 18:56:57.083722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.083684 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-monitoring-tls-certs\") pod \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\" (UID: \"4031a1c8-25ce-46fd-8f39-99d17c9d2f43\") " Oct 30 18:56:57.084148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.084101 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otc-internal" pod "4031a1c8-25ce-46fd-8f39-99d17c9d2f43" (UID: "4031a1c8-25ce-46fd-8f39-99d17c9d2f43"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:57.086522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.086488 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otlp-tls-certs" pod "4031a1c8-25ce-46fd-8f39-99d17c9d2f43" (UID: "4031a1c8-25ce-46fd-8f39-99d17c9d2f43"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:57.086634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.086519 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-kube-api-access-hmjpb" pod "4031a1c8-25ce-46fd-8f39-99d17c9d2f43" (UID: "4031a1c8-25ce-46fd-8f39-99d17c9d2f43"). InnerVolumeSpecName "kube-api-access-hmjpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:57.086799 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.086752 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-data-science-collector-ta-client-cert" pod "4031a1c8-25ce-46fd-8f39-99d17c9d2f43" (UID: "4031a1c8-25ce-46fd-8f39-99d17c9d2f43"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:57.086799 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.086787 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-tls-certs" pod "4031a1c8-25ce-46fd-8f39-99d17c9d2f43" (UID: "4031a1c8-25ce-46fd-8f39-99d17c9d2f43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:57.086917 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.086775 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-monitoring-tls-certs" pod "4031a1c8-25ce-46fd-8f39-99d17c9d2f43" (UID: "4031a1c8-25ce-46fd-8f39-99d17c9d2f43"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:57.185146 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.185124 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:57.185146 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.185145 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:57.185277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.185156 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmjpb\" (UniqueName: \"kubernetes.io/projected/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-kube-api-access-hmjpb\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:57.185277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.185170 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:57.185277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.185182 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:57.185277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.185189 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4031a1c8-25ce-46fd-8f39-99d17c9d2f43-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:56:57.750611 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.750546 5014 generic.go:356] "Generic (PLEG): container finished" podID="4031a1c8-25ce-46fd-8f39-99d17c9d2f43" containerID="61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb" exitCode=0 Oct 30 18:56:57.750769 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.750625 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:57.750769 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.750638 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4031a1c8-25ce-46fd-8f39-99d17c9d2f43","Type":"ContainerDied","Data":"61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb"} Oct 30 18:56:57.750769 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.750734 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4031a1c8-25ce-46fd-8f39-99d17c9d2f43","Type":"ContainerDied","Data":"d30ea0e17bacca3697dd4f27a541d36bd8febee53bd5d258faa8681d8cb44b17"} Oct 30 18:56:57.750769 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.750765 5014 scope.go:122] "RemoveContainer" containerID="61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb" Oct 30 18:56:57.760709 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.760683 5014 scope.go:122] "RemoveContainer" containerID="61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb" Oct 30 18:56:57.761017 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:56:57.760996 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb\": container with ID starting with 61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb not found: ID does not exist" containerID="61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb" Oct 30 18:56:57.761139 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.761026 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb"} err="failed to get container status \"61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb\": rpc error: code = NotFound desc = could not find container \"61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb\": container with ID starting with 61ead844ed90b9d159a294e8f77a7ea4854748a28ab72e629995236cd36e60eb not found: ID does not exist" Oct 30 18:56:57.800857 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.800831 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:57.812205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.812185 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:57.851045 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.851021 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:57.851484 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.851469 5014 cpu_manager.go:414] "containerMap: removing container" podUID="4031a1c8-25ce-46fd-8f39-99d17c9d2f43" containerName="otc-container" Oct 30 18:56:57.851552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.851486 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4031a1c8-25ce-46fd-8f39-99d17c9d2f43" containerName="otc-container" Oct 30 18:56:57.851604 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.851579 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="4031a1c8-25ce-46fd-8f39-99d17c9d2f43" containerName="otc-container" Oct 30 18:56:57.855932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.855916 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:57.870470 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.870429 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:56:57.870470 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.870428 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:56:57.870470 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.870470 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:56:57.870681 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.870488 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-p8bgw\"" Oct 30 18:56:57.870681 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.870544 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:56:57.870681 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.870666 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:56:57.883889 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.883867 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:57.991434 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.991403 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:57.991865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.991447 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:57.991865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.991493 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:57.991865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.991586 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:57.991865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.991640 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:57.991865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:57.991742 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjn62\" (UniqueName: \"kubernetes.io/projected/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-kube-api-access-gjn62\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.092524 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.092502 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjn62\" (UniqueName: \"kubernetes.io/projected/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-kube-api-access-gjn62\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.092665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.092536 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.092665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.092615 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.092665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.092655 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.092836 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.092689 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.092836 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.092719 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.093476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.093452 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.095267 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.095247 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.095412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.095397 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.095874 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.095859 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.096080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.096060 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.115970 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.115950 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjn62\" (UniqueName: \"kubernetes.io/projected/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-kube-api-access-gjn62\") pod \"data-science-collector-collector-0\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.168014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.167993 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:56:58.465645 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.465558 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:56:58.469659 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:56:58.469632 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cc9e9c_d525_43ca_a7b4_7e8c58065e5a.slice/crio-818d0c4e2b5f4168624b8cf5f991e0943b19264c2a0c64383fe6d074844c4397 WatchSource:0}: Error finding container 818d0c4e2b5f4168624b8cf5f991e0943b19264c2a0c64383fe6d074844c4397: Status 404 returned error can't find the container with id 818d0c4e2b5f4168624b8cf5f991e0943b19264c2a0c64383fe6d074844c4397 Oct 30 18:56:58.471872 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.471816 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:56:58.471960 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.471911 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:56:58.758078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.757988 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a","Type":"ContainerStarted","Data":"46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2"} Oct 30 18:56:58.758078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.758047 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a","Type":"ContainerStarted","Data":"818d0c4e2b5f4168624b8cf5f991e0943b19264c2a0c64383fe6d074844c4397"} Oct 30 18:56:58.802151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:58.802108 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.802096416 podStartE2EDuration="1.802096416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:58.800093972 +0000 UTC m=+3631.788434464" watchObservedRunningTime="2026-10-30 18:56:58.802096416 +0000 UTC m=+3631.790436935" Oct 30 18:56:59.141077 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:56:59.141047 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4031a1c8-25ce-46fd-8f39-99d17c9d2f43" path="/var/lib/kubelet/pods/4031a1c8-25ce-46fd-8f39-99d17c9d2f43/volumes" Oct 30 18:57:13.333949 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:13.333867 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:57:14.816804 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:14.816313 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:14.816804 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:14.816541 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" containerName="otc-container" containerID="cri-o://46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2" gracePeriod=30 Oct 30 18:57:17.168807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.168776 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:17.349102 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.349073 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-kube-api-access-gjn62\" (UniqueName: \"kubernetes.io/projected/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-kube-api-access-gjn62\") pod \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " Oct 30 18:57:17.349271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.349127 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otlp-tls-certs\") pod \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " Oct 30 18:57:17.349271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.349168 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-data-science-collector-ta-client-cert\") pod \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " Oct 30 18:57:17.349271 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.349199 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-tls-certs\") pod \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " Oct 30 18:57:17.349446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.349332 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-monitoring-tls-certs\") pod \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " Oct 30 18:57:17.349446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.349406 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otc-internal\") pod \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\" (UID: \"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a\") " Oct 30 18:57:17.349857 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.349827 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otc-internal" pod "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" (UID: "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:17.351473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.351452 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-tls-certs" pod "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" (UID: "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:17.351950 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.351926 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-data-science-collector-ta-client-cert" pod "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" (UID: "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:17.352032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.351969 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-kube-api-access-gjn62" pod "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" (UID: "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a"). InnerVolumeSpecName "kube-api-access-gjn62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:17.352077 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.352048 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otlp-tls-certs" pod "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" (UID: "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:17.352077 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.352064 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-monitoring-tls-certs" pod "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" (UID: "b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:17.450810 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.450786 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjn62\" (UniqueName: \"kubernetes.io/projected/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-kube-api-access-gjn62\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:17.450810 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.450806 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:17.450945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.450818 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:17.450945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.450827 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:17.450945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.450838 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:17.450945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.450845 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:17.864357 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.864309 5014 generic.go:356] "Generic (PLEG): container finished" podID="b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" containerID="46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2" exitCode=0 Oct 30 18:57:17.864552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.864371 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a","Type":"ContainerDied","Data":"46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2"} Oct 30 18:57:17.864552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.864391 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:17.864552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.864455 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a","Type":"ContainerDied","Data":"818d0c4e2b5f4168624b8cf5f991e0943b19264c2a0c64383fe6d074844c4397"} Oct 30 18:57:17.864552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.864488 5014 scope.go:122] "RemoveContainer" containerID="46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2" Oct 30 18:57:17.874925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.874904 5014 scope.go:122] "RemoveContainer" containerID="46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2" Oct 30 18:57:17.875235 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:17.875215 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2\": container with ID starting with 46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2 not found: ID does not exist" containerID="46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2" Oct 30 18:57:17.875285 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.875246 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2"} err="failed to get container status \"46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2\": rpc error: code = NotFound desc = could not find container \"46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2\": container with ID starting with 46cb0de994fd5329e1d0c3097f2a52108e4f7b3d6fdd5cf62fca6a2a958334a2 not found: ID does not exist" Oct 30 18:57:17.914214 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.914191 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:17.925737 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.925711 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:17.954403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.954371 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:17.954789 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.954775 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" containerName="otc-container" Oct 30 18:57:17.954868 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.954790 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" containerName="otc-container" Oct 30 18:57:17.954907 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.954883 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" containerName="otc-container" Oct 30 18:57:17.959394 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.959376 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:17.973938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.973920 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:57:17.973938 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.973921 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:57:17.974069 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.973921 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:57:17.974069 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.973938 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-p8bgw\"" Oct 30 18:57:17.974413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.974391 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:57:17.974650 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.974637 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:57:17.986823 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:17.986804 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:18.056244 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.056221 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.056342 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.056253 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.056342 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.056273 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/27c1cc4d-7c04-40c6-8ed5-d421289676af-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.056419 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.056385 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxw9\" (UniqueName: \"kubernetes.io/projected/27c1cc4d-7c04-40c6-8ed5-d421289676af-kube-api-access-bfxw9\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.056458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.056450 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.056523 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.056507 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.157084 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.157029 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.157084 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.157061 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.157216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.157091 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.157216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.157127 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.157216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.157158 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/27c1cc4d-7c04-40c6-8ed5-d421289676af-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.157381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.157304 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxw9\" (UniqueName: \"kubernetes.io/projected/27c1cc4d-7c04-40c6-8ed5-d421289676af-kube-api-access-bfxw9\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.158112 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.158087 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/27c1cc4d-7c04-40c6-8ed5-d421289676af-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.159722 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.159696 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.159883 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.159861 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.159928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.159874 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.159928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.159901 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.180103 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.180085 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxw9\" (UniqueName: \"kubernetes.io/projected/27c1cc4d-7c04-40c6-8ed5-d421289676af-kube-api-access-bfxw9\") pod \"data-science-collector-collector-0\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.272072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.272054 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:18.568164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.568136 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:18.572132 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:57:18.572103 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c1cc4d_7c04_40c6_8ed5_d421289676af.slice/crio-b968d1eac2223d5203ece6a5895716a52836287d2846f1501f4be04fc53e1386 WatchSource:0}: Error finding container b968d1eac2223d5203ece6a5895716a52836287d2846f1501f4be04fc53e1386: Status 404 returned error can't find the container with id b968d1eac2223d5203ece6a5895716a52836287d2846f1501f4be04fc53e1386 Oct 30 18:57:18.574294 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.574263 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:57:18.574377 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.574321 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:57:18.871019 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.870927 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"27c1cc4d-7c04-40c6-8ed5-d421289676af","Type":"ContainerStarted","Data":"66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1"} Oct 30 18:57:18.871019 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.870976 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"27c1cc4d-7c04-40c6-8ed5-d421289676af","Type":"ContainerStarted","Data":"b968d1eac2223d5203ece6a5895716a52836287d2846f1501f4be04fc53e1386"} Oct 30 18:57:18.914514 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:18.914446 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.9144335350000001 podStartE2EDuration="1.914433535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:18.912523928 +0000 UTC m=+3651.900864414" watchObservedRunningTime="2026-10-30 18:57:18.914433535 +0000 UTC m=+3651.902774023" Oct 30 18:57:19.053183 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.053149 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx"] Oct 30 18:57:19.053427 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.053402 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" podUID="63ea83bc-7552-4cc1-805f-9f5eb6d10513" containerName="kube-rbac-proxy" containerID="cri-o://677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5" gracePeriod=30 Oct 30 18:57:19.067983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.067962 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw"] Oct 30 18:57:19.068194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.068169 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="method-gate" containerID="cri-o://8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44" gracePeriod=30 Oct 30 18:57:19.068306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.068168 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="prom-label-proxy" containerID="cri-o://4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d" gracePeriod=30 Oct 30 18:57:19.068413 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.068177 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="kube-rbac-proxy" containerID="cri-o://082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5" gracePeriod=30 Oct 30 18:57:19.142004 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.141979 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a" path="/var/lib/kubelet/pods/b2cc9e9c-d525-43ca-a7b4-7e8c58065e5a/volumes" Oct 30 18:57:19.319151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.319129 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:57:19.352332 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.352304 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:57:19.393485 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.393406 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8"] Oct 30 18:57:19.393695 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.393606 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" podUID="f2e66b4d-3b28-4974-84e8-c1a31773f2c4" containerName="thanos-querier" containerID="cri-o://1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22" gracePeriod=30 Oct 30 18:57:19.469624 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.469577 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-rbac-proxy-config\") pod \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " Oct 30 18:57:19.469802 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.469638 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-config\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-config\") pod \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " Oct 30 18:57:19.469802 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.469701 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-tls\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-tls\") pod \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " Oct 30 18:57:19.469802 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.469729 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-prometheus-tls-ca\") pod \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " Oct 30 18:57:19.470026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.470000 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-config" pod "ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" (UID: "ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:19.470142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.470123 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-tls\" (UniqueName: \"kubernetes.io/secret/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-tls\") pod \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " Oct 30 18:57:19.470268 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.470248 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-kube-api-access-7x4b5\" (UniqueName: \"kubernetes.io/projected/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-kube-api-access-7x4b5\") pod \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " Oct 30 18:57:19.470557 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.470295 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-prometheus-tls-ca" pod "ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" (UID: "ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:19.470826 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.470805 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-client-ca\") pod \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\" (UID: \"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e\") " Oct 30 18:57:19.470964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.470949 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-ca\") pod \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " Oct 30 18:57:19.471094 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.471077 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-api-access-4htjd\" (UniqueName: \"kubernetes.io/projected/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-api-access-4htjd\") pod \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " Oct 30 18:57:19.471208 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.471194 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-client-cert\") pod \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\" (UID: \"63ea83bc-7552-4cc1-805f-9f5eb6d10513\") " Oct 30 18:57:19.471546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.471519 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-ca" pod "63ea83bc-7552-4cc1-805f-9f5eb6d10513" (UID: "63ea83bc-7552-4cc1-805f-9f5eb6d10513"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:19.471751 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.471733 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.472134 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.472119 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.472355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.472340 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-prometheus-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.472458 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.472031 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-client-ca" pod "ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" (UID: "ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:19.472770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.472289 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-tls" pod "63ea83bc-7552-4cc1-805f-9f5eb6d10513" (UID: "63ea83bc-7552-4cc1-805f-9f5eb6d10513"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:19.472874 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.472815 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-tls" pod "ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" (UID: "ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:19.472997 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.472969 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-rbac-proxy-config" pod "63ea83bc-7552-4cc1-805f-9f5eb6d10513" (UID: "63ea83bc-7552-4cc1-805f-9f5eb6d10513"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:19.473411 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.473384 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-kube-api-access-7x4b5" pod "ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" (UID: "ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e"). InnerVolumeSpecName "kube-api-access-7x4b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:19.473998 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.473976 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-client-cert" pod "63ea83bc-7552-4cc1-805f-9f5eb6d10513" (UID: "63ea83bc-7552-4cc1-805f-9f5eb6d10513"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:19.474473 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.474455 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-api-access-4htjd" pod "63ea83bc-7552-4cc1-805f-9f5eb6d10513" (UID: "63ea83bc-7552-4cc1-805f-9f5eb6d10513"). InnerVolumeSpecName "kube-api-access-4htjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:19.564983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.564157 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:57:19.564983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.564498 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="prometheus" containerID="cri-o://d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b" gracePeriod=600 Oct 30 18:57:19.564983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.564645 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="thanos-sidecar" containerID="cri-o://3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513" gracePeriod=600 Oct 30 18:57:19.564983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.564709 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="config-reloader" containerID="cri-o://cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1" gracePeriod=600 Oct 30 18:57:19.574758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.574728 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4htjd\" (UniqueName: \"kubernetes.io/projected/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-api-access-4htjd\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.574852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.574752 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-prometheus-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.574852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.574778 5014 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-kube-rbac-proxy-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.574852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.574788 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/63ea83bc-7552-4cc1-805f-9f5eb6d10513-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.574852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.574798 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.574852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.574812 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x4b5\" (UniqueName: \"kubernetes.io/projected/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-kube-api-access-7x4b5\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.574852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.574826 5014 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e-client-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.575922 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.575900 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:57:19.576221 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.576185 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="alertmanager" containerID="cri-o://427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421" gracePeriod=120 Oct 30 18:57:19.577093 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.576376 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="config-reloader" containerID="cri-o://651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8" gracePeriod=120 Oct 30 18:57:19.690971 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.690950 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" Oct 30 18:57:19.708870 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.708838 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:57:19.776039 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.776013 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e66b4d-3b28-4974-84e8-c1a31773f2c4-kube-api-access-2rlbl\" (UniqueName: \"kubernetes.io/projected/f2e66b4d-3b28-4974-84e8-c1a31773f2c4-kube-api-access-2rlbl\") pod \"f2e66b4d-3b28-4974-84e8-c1a31773f2c4\" (UID: \"f2e66b4d-3b28-4974-84e8-c1a31773f2c4\") " Oct 30 18:57:19.778184 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.778156 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e66b4d-3b28-4974-84e8-c1a31773f2c4-kube-api-access-2rlbl" pod "f2e66b4d-3b28-4974-84e8-c1a31773f2c4" (UID: "f2e66b4d-3b28-4974-84e8-c1a31773f2c4"). InnerVolumeSpecName "kube-api-access-2rlbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:19.876680 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.876652 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rlbl\" (UniqueName: \"kubernetes.io/projected/f2e66b4d-3b28-4974-84e8-c1a31773f2c4-kube-api-access-2rlbl\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.878865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.878841 5014 generic.go:356] "Generic (PLEG): container finished" podID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerID="3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513" exitCode=0 Oct 30 18:57:19.878865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.878863 5014 generic.go:356] "Generic (PLEG): container finished" podID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerID="cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1" exitCode=0 Oct 30 18:57:19.879052 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.878912 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"116e8cd8-5f76-4b30-a446-0051cac4a571","Type":"ContainerDied","Data":"3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513"} Oct 30 18:57:19.879052 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.878996 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"116e8cd8-5f76-4b30-a446-0051cac4a571","Type":"ContainerDied","Data":"cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1"} Oct 30 18:57:19.880355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.880334 5014 generic.go:356] "Generic (PLEG): container finished" podID="f2e66b4d-3b28-4974-84e8-c1a31773f2c4" containerID="1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22" exitCode=0 Oct 30 18:57:19.880457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.880380 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" Oct 30 18:57:19.880457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.880387 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" event={"ID":"f2e66b4d-3b28-4974-84e8-c1a31773f2c4","Type":"ContainerDied","Data":"1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22"} Oct 30 18:57:19.880457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.880423 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8" event={"ID":"f2e66b4d-3b28-4974-84e8-c1a31773f2c4","Type":"ContainerDied","Data":"4b39790664e5cb66e006ade991731eea631b9441902efe1635edbd3334f1f1df"} Oct 30 18:57:19.880457 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.880445 5014 scope.go:122] "RemoveContainer" containerID="1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22" Oct 30 18:57:19.882749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.882726 5014 generic.go:356] "Generic (PLEG): container finished" podID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerID="4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d" exitCode=0 Oct 30 18:57:19.882749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.882746 5014 generic.go:356] "Generic (PLEG): container finished" podID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerID="8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44" exitCode=0 Oct 30 18:57:19.882749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.882753 5014 generic.go:356] "Generic (PLEG): container finished" podID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerID="082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5" exitCode=0 Oct 30 18:57:19.883006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.882790 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" Oct 30 18:57:19.883006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.882807 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" event={"ID":"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e","Type":"ContainerDied","Data":"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d"} Oct 30 18:57:19.883006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.882883 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" event={"ID":"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e","Type":"ContainerDied","Data":"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44"} Oct 30 18:57:19.883006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.882924 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" event={"ID":"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e","Type":"ContainerDied","Data":"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5"} Oct 30 18:57:19.883006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.882962 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw" event={"ID":"ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e","Type":"ContainerDied","Data":"30c1c129ee366b62d01fc14ad5486038cd3d2c658862835db10b21528d0442b3"} Oct 30 18:57:19.884491 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.884469 5014 generic.go:356] "Generic (PLEG): container finished" podID="63ea83bc-7552-4cc1-805f-9f5eb6d10513" containerID="677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5" exitCode=0 Oct 30 18:57:19.884623 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.884531 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" Oct 30 18:57:19.884623 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.884551 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" event={"ID":"63ea83bc-7552-4cc1-805f-9f5eb6d10513","Type":"ContainerDied","Data":"677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5"} Oct 30 18:57:19.884884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.884622 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx" event={"ID":"63ea83bc-7552-4cc1-805f-9f5eb6d10513","Type":"ContainerDied","Data":"578374ba907bc82b90bf280452133d5f85a26cbd31f558d6dfd4a0b91941d0f3"} Oct 30 18:57:19.890211 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.890169 5014 scope.go:122] "RemoveContainer" containerID="1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22" Oct 30 18:57:19.890468 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:19.890441 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22\": container with ID starting with 1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22 not found: ID does not exist" containerID="1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22" Oct 30 18:57:19.890629 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.890479 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22"} err="failed to get container status \"1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22\": rpc error: code = NotFound desc = could not find container \"1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22\": container with ID starting with 1e09d0fdd277702dd99d863576d041c8abfee4f6028de5d0ce7e7cfd0fd04c22 not found: ID does not exist" Oct 30 18:57:19.890629 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.890502 5014 scope.go:122] "RemoveContainer" containerID="4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d" Oct 30 18:57:19.890996 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.890974 5014 generic.go:356] "Generic (PLEG): container finished" podID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerID="651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8" exitCode=0 Oct 30 18:57:19.891114 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.891001 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"badf1802-ffc1-4649-82a8-5a71b2827a01","Type":"ContainerDied","Data":"651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8"} Oct 30 18:57:19.900364 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.900349 5014 scope.go:122] "RemoveContainer" containerID="8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44" Oct 30 18:57:19.908224 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.908204 5014 scope.go:122] "RemoveContainer" containerID="082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5" Oct 30 18:57:19.915889 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.915873 5014 scope.go:122] "RemoveContainer" containerID="4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d" Oct 30 18:57:19.916143 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:19.916124 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d\": container with ID starting with 4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d not found: ID does not exist" containerID="4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d" Oct 30 18:57:19.916203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.916148 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d"} err="failed to get container status \"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d\": rpc error: code = NotFound desc = could not find container \"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d\": container with ID starting with 4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d not found: ID does not exist" Oct 30 18:57:19.916203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.916162 5014 scope.go:122] "RemoveContainer" containerID="8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44" Oct 30 18:57:19.916448 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:19.916430 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44\": container with ID starting with 8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44 not found: ID does not exist" containerID="8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44" Oct 30 18:57:19.916498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.916453 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44"} err="failed to get container status \"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44\": rpc error: code = NotFound desc = could not find container \"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44\": container with ID starting with 8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44 not found: ID does not exist" Oct 30 18:57:19.916498 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.916468 5014 scope.go:122] "RemoveContainer" containerID="082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5" Oct 30 18:57:19.916724 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:19.916704 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5\": container with ID starting with 082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5 not found: ID does not exist" containerID="082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5" Oct 30 18:57:19.916819 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.916733 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5"} err="failed to get container status \"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5\": rpc error: code = NotFound desc = could not find container \"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5\": container with ID starting with 082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5 not found: ID does not exist" Oct 30 18:57:19.916819 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.916753 5014 scope.go:122] "RemoveContainer" containerID="4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d" Oct 30 18:57:19.916985 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.916967 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d"} err="failed to get container status \"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d\": rpc error: code = NotFound desc = could not find container \"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d\": container with ID starting with 4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d not found: ID does not exist" Oct 30 18:57:19.917031 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.916985 5014 scope.go:122] "RemoveContainer" containerID="8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44" Oct 30 18:57:19.917231 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.917213 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44"} err="failed to get container status \"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44\": rpc error: code = NotFound desc = could not find container \"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44\": container with ID starting with 8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44 not found: ID does not exist" Oct 30 18:57:19.917231 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.917232 5014 scope.go:122] "RemoveContainer" containerID="082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5" Oct 30 18:57:19.917463 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.917446 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5"} err="failed to get container status \"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5\": rpc error: code = NotFound desc = could not find container \"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5\": container with ID starting with 082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5 not found: ID does not exist" Oct 30 18:57:19.917525 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.917463 5014 scope.go:122] "RemoveContainer" containerID="4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d" Oct 30 18:57:19.917710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.917692 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d"} err="failed to get container status \"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d\": rpc error: code = NotFound desc = could not find container \"4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d\": container with ID starting with 4a31ec9efe1fa7f702f42248f84f6954c0d8ad9d943b3eba21793ab75f5e370d not found: ID does not exist" Oct 30 18:57:19.917710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.917710 5014 scope.go:122] "RemoveContainer" containerID="8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44" Oct 30 18:57:19.917925 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.917907 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44"} err="failed to get container status \"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44\": rpc error: code = NotFound desc = could not find container \"8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44\": container with ID starting with 8eda516fd729c9a60042651449a82c11394a6cae08af87d785d33c7dc281ce44 not found: ID does not exist" Oct 30 18:57:19.917986 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.917925 5014 scope.go:122] "RemoveContainer" containerID="082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5" Oct 30 18:57:19.918150 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.918135 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5"} err="failed to get container status \"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5\": rpc error: code = NotFound desc = could not find container \"082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5\": container with ID starting with 082f371d7a46311d282411b37c6a5c915222b5d33049cee9cd96ab292fcaf8c5 not found: ID does not exist" Oct 30 18:57:19.918150 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.918149 5014 scope.go:122] "RemoveContainer" containerID="677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5" Oct 30 18:57:19.925742 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.925727 5014 scope.go:122] "RemoveContainer" containerID="677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5" Oct 30 18:57:19.926014 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:19.925995 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5\": container with ID starting with 677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5 not found: ID does not exist" containerID="677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5" Oct 30 18:57:19.926066 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.926019 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5"} err="failed to get container status \"677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5\": rpc error: code = NotFound desc = could not find container \"677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5\": container with ID starting with 677b491cd0c43771c72c8d28b98185a6e3899809c9042e2ec33626d7514fc4c5 not found: ID does not exist" Oct 30 18:57:19.945227 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.945183 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8"] Oct 30 18:57:19.949559 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.949537 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-c9mq8"] Oct 30 18:57:19.997014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:19.996987 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx"] Oct 30 18:57:20.004500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:20.004482 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-65lcx"] Oct 30 18:57:20.039785 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:20.039765 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw"] Oct 30 18:57:20.048139 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:20.048121 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-zkrsw"] Oct 30 18:57:20.207357 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:20.207286 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:20.899364 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:20.899327 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="27c1cc4d-7c04-40c6-8ed5-d421289676af" containerName="otc-container" containerID="cri-o://66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1" gracePeriod=30 Oct 30 18:57:21.148026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.147993 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ea83bc-7552-4cc1-805f-9f5eb6d10513" path="/var/lib/kubelet/pods/63ea83bc-7552-4cc1-805f-9f5eb6d10513/volumes" Oct 30 18:57:21.148554 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.148532 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" path="/var/lib/kubelet/pods/ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e/volumes" Oct 30 18:57:21.149316 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.149293 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e66b4d-3b28-4974-84e8-c1a31773f2c4" path="/var/lib/kubelet/pods/f2e66b4d-3b28-4974-84e8-c1a31773f2c4/volumes" Oct 30 18:57:21.199982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.199959 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:21.289230 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.289204 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-tls-certs\") pod \"27c1cc4d-7c04-40c6-8ed5-d421289676af\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " Oct 30 18:57:21.289378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.289239 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c1cc4d-7c04-40c6-8ed5-d421289676af-otc-internal\" (UniqueName: \"kubernetes.io/configmap/27c1cc4d-7c04-40c6-8ed5-d421289676af-otc-internal\") pod \"27c1cc4d-7c04-40c6-8ed5-d421289676af\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " Oct 30 18:57:21.289378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.289264 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-monitoring-tls-certs\") pod \"27c1cc4d-7c04-40c6-8ed5-d421289676af\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " Oct 30 18:57:21.289378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.289317 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c1cc4d-7c04-40c6-8ed5-d421289676af-kube-api-access-bfxw9\" (UniqueName: \"kubernetes.io/projected/27c1cc4d-7c04-40c6-8ed5-d421289676af-kube-api-access-bfxw9\") pod \"27c1cc4d-7c04-40c6-8ed5-d421289676af\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " Oct 30 18:57:21.289378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.289372 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-data-science-collector-ta-client-cert\") pod \"27c1cc4d-7c04-40c6-8ed5-d421289676af\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " Oct 30 18:57:21.289618 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.289403 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-otlp-tls-certs\") pod \"27c1cc4d-7c04-40c6-8ed5-d421289676af\" (UID: \"27c1cc4d-7c04-40c6-8ed5-d421289676af\") " Oct 30 18:57:21.289739 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.289706 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c1cc4d-7c04-40c6-8ed5-d421289676af-otc-internal" pod "27c1cc4d-7c04-40c6-8ed5-d421289676af" (UID: "27c1cc4d-7c04-40c6-8ed5-d421289676af"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:21.291691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.291661 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c1cc4d-7c04-40c6-8ed5-d421289676af-kube-api-access-bfxw9" pod "27c1cc4d-7c04-40c6-8ed5-d421289676af" (UID: "27c1cc4d-7c04-40c6-8ed5-d421289676af"). InnerVolumeSpecName "kube-api-access-bfxw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:21.291816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.291735 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-monitoring-tls-certs" pod "27c1cc4d-7c04-40c6-8ed5-d421289676af" (UID: "27c1cc4d-7c04-40c6-8ed5-d421289676af"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:21.291816 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.291760 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-otlp-tls-certs" pod "27c1cc4d-7c04-40c6-8ed5-d421289676af" (UID: "27c1cc4d-7c04-40c6-8ed5-d421289676af"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:21.291948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.291930 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-data-science-collector-ta-client-cert" pod "27c1cc4d-7c04-40c6-8ed5-d421289676af" (UID: "27c1cc4d-7c04-40c6-8ed5-d421289676af"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:21.292041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.292025 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-tls-certs" pod "27c1cc4d-7c04-40c6-8ed5-d421289676af" (UID: "27c1cc4d-7c04-40c6-8ed5-d421289676af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:21.390181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.390151 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:21.390181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.390177 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfxw9\" (UniqueName: \"kubernetes.io/projected/27c1cc4d-7c04-40c6-8ed5-d421289676af-kube-api-access-bfxw9\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:21.390318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.390193 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:21.390318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.390207 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:21.390318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.390221 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c1cc4d-7c04-40c6-8ed5-d421289676af-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:21.390318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.390233 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/27c1cc4d-7c04-40c6-8ed5-d421289676af-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:21.638991 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.638961 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.153:9093/-/ready\": dial tcp 10.132.0.153:9093: connect: connection refused" Oct 30 18:57:21.905369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.905289 5014 generic.go:356] "Generic (PLEG): container finished" podID="27c1cc4d-7c04-40c6-8ed5-d421289676af" containerID="66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1" exitCode=0 Oct 30 18:57:21.905369 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.905342 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:21.905833 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.905379 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"27c1cc4d-7c04-40c6-8ed5-d421289676af","Type":"ContainerDied","Data":"66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1"} Oct 30 18:57:21.905833 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.905429 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"27c1cc4d-7c04-40c6-8ed5-d421289676af","Type":"ContainerDied","Data":"b968d1eac2223d5203ece6a5895716a52836287d2846f1501f4be04fc53e1386"} Oct 30 18:57:21.905833 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.905450 5014 scope.go:122] "RemoveContainer" containerID="66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1" Oct 30 18:57:21.915170 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.915155 5014 scope.go:122] "RemoveContainer" containerID="66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1" Oct 30 18:57:21.915424 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:21.915403 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1\": container with ID starting with 66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1 not found: ID does not exist" containerID="66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1" Oct 30 18:57:21.915485 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.915437 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1"} err="failed to get container status \"66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1\": rpc error: code = NotFound desc = could not find container \"66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1\": container with ID starting with 66e7c0031d313d5146f957bdcf9098e162f894544286e51a392d7c472877cbf1 not found: ID does not exist" Oct 30 18:57:21.954813 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.954784 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:21.966493 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:21.966471 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:23.141690 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:23.141655 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c1cc4d-7c04-40c6-8ed5-d421289676af" path="/var/lib/kubelet/pods/27c1cc4d-7c04-40c6-8ed5-d421289676af/volumes" Oct 30 18:57:24.708632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:24.708574 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:57:26.639382 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:26.639336 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.153:9093/-/ready\": dial tcp 10.132.0.153:9093: connect: connection refused" Oct 30 18:57:29.709002 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.708971 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:57:29.709343 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.709119 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:29.826772 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.826749 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:29.854151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.854095 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-alertmanager-data-science-monitoringstack-db\") pod \"badf1802-ffc1-4649-82a8-5a71b2827a01\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " Oct 30 18:57:29.854151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.854141 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-cluster-tls-config\") pod \"badf1802-ffc1-4649-82a8-5a71b2827a01\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " Oct 30 18:57:29.854381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.854194 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-tls-assets\" (UniqueName: \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-tls-assets\") pod \"badf1802-ffc1-4649-82a8-5a71b2827a01\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " Oct 30 18:57:29.854381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.854241 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-config-volume\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-config-volume\") pod \"badf1802-ffc1-4649-82a8-5a71b2827a01\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " Oct 30 18:57:29.854381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.854276 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-kube-api-access-ccvxk\" (UniqueName: \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-kube-api-access-ccvxk\") pod \"badf1802-ffc1-4649-82a8-5a71b2827a01\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " Oct 30 18:57:29.854381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.854307 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-config-out\" (UniqueName: \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-config-out\") pod \"badf1802-ffc1-4649-82a8-5a71b2827a01\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " Oct 30 18:57:29.854589 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.854389 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-web-config\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-web-config\") pod \"badf1802-ffc1-4649-82a8-5a71b2827a01\" (UID: \"badf1802-ffc1-4649-82a8-5a71b2827a01\") " Oct 30 18:57:29.854589 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.854406 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-alertmanager-data-science-monitoringstack-db" pod "badf1802-ffc1-4649-82a8-5a71b2827a01" (UID: "badf1802-ffc1-4649-82a8-5a71b2827a01"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:29.854732 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.854666 5014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.856649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.856577 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-config-out" pod "badf1802-ffc1-4649-82a8-5a71b2827a01" (UID: "badf1802-ffc1-4649-82a8-5a71b2827a01"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:29.856649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.856624 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-kube-api-access-ccvxk" pod "badf1802-ffc1-4649-82a8-5a71b2827a01" (UID: "badf1802-ffc1-4649-82a8-5a71b2827a01"). InnerVolumeSpecName "kube-api-access-ccvxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:29.857007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.856974 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-config-volume" pod "badf1802-ffc1-4649-82a8-5a71b2827a01" (UID: "badf1802-ffc1-4649-82a8-5a71b2827a01"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:29.857122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.857029 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-tls-assets" pod "badf1802-ffc1-4649-82a8-5a71b2827a01" (UID: "badf1802-ffc1-4649-82a8-5a71b2827a01"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:29.860848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.860828 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-cluster-tls-config" pod "badf1802-ffc1-4649-82a8-5a71b2827a01" (UID: "badf1802-ffc1-4649-82a8-5a71b2827a01"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:29.868576 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.868554 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-web-config" pod "badf1802-ffc1-4649-82a8-5a71b2827a01" (UID: "badf1802-ffc1-4649-82a8-5a71b2827a01"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:29.944920 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.944893 5014 generic.go:356] "Generic (PLEG): container finished" podID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerID="427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421" exitCode=0 Oct 30 18:57:29.945040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.944951 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:29.945040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.944972 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"badf1802-ffc1-4649-82a8-5a71b2827a01","Type":"ContainerDied","Data":"427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421"} Oct 30 18:57:29.945040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.945017 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"badf1802-ffc1-4649-82a8-5a71b2827a01","Type":"ContainerDied","Data":"9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec"} Oct 30 18:57:29.945179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.945043 5014 scope.go:122] "RemoveContainer" containerID="651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8" Oct 30 18:57:29.953959 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.953941 5014 scope.go:122] "RemoveContainer" containerID="427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421" Oct 30 18:57:29.955057 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.955039 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.955138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.955062 5014 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-cluster-tls-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.955138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.955075 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.955138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.955087 5014 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/badf1802-ffc1-4649-82a8-5a71b2827a01-config-volume\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.955138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.955099 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccvxk\" (UniqueName: \"kubernetes.io/projected/badf1802-ffc1-4649-82a8-5a71b2827a01-kube-api-access-ccvxk\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.955138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.955112 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/badf1802-ffc1-4649-82a8-5a71b2827a01-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.961631 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.961615 5014 scope.go:122] "RemoveContainer" containerID="f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d" Oct 30 18:57:29.968630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.968614 5014 scope.go:122] "RemoveContainer" containerID="651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8" Oct 30 18:57:29.968865 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:29.968848 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8\": container with ID starting with 651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8 not found: ID does not exist" containerID="651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8" Oct 30 18:57:29.968916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.968874 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8"} err="failed to get container status \"651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8\": rpc error: code = NotFound desc = could not find container \"651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8\": container with ID starting with 651f510ecd0aabead67c5e8fecf8256e69ce9223783b8d85428e888eeda234e8 not found: ID does not exist" Oct 30 18:57:29.968916 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.968889 5014 scope.go:122] "RemoveContainer" containerID="427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421" Oct 30 18:57:29.969118 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:29.969100 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421\": container with ID starting with 427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421 not found: ID does not exist" containerID="427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421" Oct 30 18:57:29.969178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.969127 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421"} err="failed to get container status \"427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421\": rpc error: code = NotFound desc = could not find container \"427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421\": container with ID starting with 427670c59da495a9fc243533bd96343f9ea7bf6323bfebe05769d4c83dc82421 not found: ID does not exist" Oct 30 18:57:29.969178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.969149 5014 scope.go:122] "RemoveContainer" containerID="f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d" Oct 30 18:57:29.969422 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:29.969403 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d\": container with ID starting with f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d not found: ID does not exist" containerID="f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d" Oct 30 18:57:29.969461 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.969428 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d"} err="failed to get container status \"f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d\": rpc error: code = NotFound desc = could not find container \"f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d\": container with ID starting with f2fa45f17c629b95f55c1ee9a4296fa0ddbd1e63a793305ddfccc61f5b19b45d not found: ID does not exist" Oct 30 18:57:29.995658 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:29.995638 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:57:30.000233 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:30.000210 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:57:31.141196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:31.141167 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" path="/var/lib/kubelet/pods/badf1802-ffc1-4649-82a8-5a71b2827a01/volumes" Oct 30 18:57:34.676536 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:34.676493 5014 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/cd05bacc340eb12cf97795321c3e11b57a70a96c5c5f67fb4d1e00247b0113c3/diff" to get inode usage: stat /var/lib/containers/storage/overlay/cd05bacc340eb12cf97795321c3e11b57a70a96c5c5f67fb4d1e00247b0113c3/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/redhat-ods-monitoring_alertmanager-data-science-monitoringstack-1_badf1802-ffc1-4649-82a8-5a71b2827a01/alertmanager/0.log" to get inode usage: stat /var/log/pods/redhat-ods-monitoring_alertmanager-data-science-monitoringstack-1_badf1802-ffc1-4649-82a8-5a71b2827a01/alertmanager/0.log: no such file or directory Oct 30 18:57:34.708515 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:34.708471 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:57:38.113841 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:38.113797 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf1802_ffc1_4649_82a8_5a71b2827a01.slice/crio-9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec\": RecentStats: unable to find data in memory cache]" Oct 30 18:57:38.214044 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.214023 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:38.318862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.318837 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.318983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.318896 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.318979 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319018 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-kube-api-access-qhprn\" (UniqueName: \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-kube-api-access-qhprn\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319131 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319042 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-thanos-prometheus-http-client-file\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319131 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319077 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319131 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319115 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-config\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-config\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319156 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-tls-assets\" (UniqueName: \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-tls-assets\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319187 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319239 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319271 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116e8cd8-5f76-4b30-a446-0051cac4a571-config-out\" (UniqueName: \"kubernetes.io/empty-dir/116e8cd8-5f76-4b30-a446-0051cac4a571-config-out\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319306 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319345 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-secret-prometheus-web-tls-ca\") pod \"116e8cd8-5f76-4b30-a446-0051cac4a571\" (UID: \"116e8cd8-5f76-4b30-a446-0051cac4a571\") " Oct 30 18:57:38.319532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319380 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-0" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:38.319803 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319770 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.320210 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.319940 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-2" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:38.321215 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.321191 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-1" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:38.322370 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.322343 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-thanos-prometheus-http-client-file" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:38.322683 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.322579 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-secret-prometheus-web-tls-ca" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:38.322683 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.322656 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-config" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:38.323522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.323488 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-tls-assets" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:38.323762 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.323739 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116e8cd8-5f76-4b30-a446-0051cac4a571-config-out" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:38.324043 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.324017 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:38.324274 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.324245 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:38.324688 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.324656 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-kube-api-access-qhprn" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "kube-api-access-qhprn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:38.325471 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.325448 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "pvc-c02e8472-bb1a-4974-aa89-86eef715889b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:57:38.334750 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.334729 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config" pod "116e8cd8-5f76-4b30-a446-0051cac4a571" (UID: "116e8cd8-5f76-4b30-a446-0051cac4a571"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:38.420177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420154 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhprn\" (UniqueName: \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-kube-api-access-qhprn\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.420177 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420176 5014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.420312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420187 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.420312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420196 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.420312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420205 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/116e8cd8-5f76-4b30-a446-0051cac4a571-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.420312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420214 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.420312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420223 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/116e8cd8-5f76-4b30-a446-0051cac4a571-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.420312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420231 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/116e8cd8-5f76-4b30-a446-0051cac4a571-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.420312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420239 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.420312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420247 5014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.420312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420255 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/116e8cd8-5f76-4b30-a446-0051cac4a571-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:57:38.420312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.420280 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:57:38.430758 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.430744 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c02e8472-bb1a-4974-aa89-86eef715889b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:57:38.521636 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.521582 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" Oct 30 18:57:38.998900 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.998862 5014 generic.go:356] "Generic (PLEG): container finished" podID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerID="d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b" exitCode=0 Oct 30 18:57:38.999030 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.998944 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"116e8cd8-5f76-4b30-a446-0051cac4a571","Type":"ContainerDied","Data":"d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b"} Oct 30 18:57:38.999030 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.998970 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:38.999110 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.999055 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"116e8cd8-5f76-4b30-a446-0051cac4a571","Type":"ContainerDied","Data":"88ca7dc837914e766f8bfe70f64f92b4c081d82b13856d8660dfbde13e984a7e"} Oct 30 18:57:38.999110 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:38.999091 5014 scope.go:122] "RemoveContainer" containerID="3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513" Oct 30 18:57:39.009439 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.009421 5014 scope.go:122] "RemoveContainer" containerID="cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1" Oct 30 18:57:39.018339 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.018319 5014 scope.go:122] "RemoveContainer" containerID="d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b" Oct 30 18:57:39.027027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.026998 5014 scope.go:122] "RemoveContainer" containerID="f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce" Oct 30 18:57:39.035192 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.035171 5014 scope.go:122] "RemoveContainer" containerID="3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513" Oct 30 18:57:39.035462 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:39.035442 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513\": container with ID starting with 3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513 not found: ID does not exist" containerID="3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513" Oct 30 18:57:39.035511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.035468 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513"} err="failed to get container status \"3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513\": rpc error: code = NotFound desc = could not find container \"3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513\": container with ID starting with 3511a8e7d7982edefd8978d9ac79399a477e032883ca41ed84679723a86de513 not found: ID does not exist" Oct 30 18:57:39.035511 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.035486 5014 scope.go:122] "RemoveContainer" containerID="cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1" Oct 30 18:57:39.035746 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:39.035727 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1\": container with ID starting with cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1 not found: ID does not exist" containerID="cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1" Oct 30 18:57:39.035795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.035752 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1"} err="failed to get container status \"cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1\": rpc error: code = NotFound desc = could not find container \"cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1\": container with ID starting with cfa39a4c5a44f04514421bfcb61dd7f31f03d0f1628e2e0244a03c821d195df1 not found: ID does not exist" Oct 30 18:57:39.035795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.035765 5014 scope.go:122] "RemoveContainer" containerID="d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b" Oct 30 18:57:39.035981 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:39.035965 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b\": container with ID starting with d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b not found: ID does not exist" containerID="d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b" Oct 30 18:57:39.036026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.035986 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b"} err="failed to get container status \"d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b\": rpc error: code = NotFound desc = could not find container \"d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b\": container with ID starting with d0fc15e933a6eda79ac0e5c7524bb380c76aa67053b9e1af35458cba3f3d220b not found: ID does not exist" Oct 30 18:57:39.036026 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.036000 5014 scope.go:122] "RemoveContainer" containerID="f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce" Oct 30 18:57:39.036206 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:39.036190 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce\": container with ID starting with f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce not found: ID does not exist" containerID="f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce" Oct 30 18:57:39.036254 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.036208 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce"} err="failed to get container status \"f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce\": rpc error: code = NotFound desc = could not find container \"f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce\": container with ID starting with f24f83efcd8f94d074389098ec6fbf13e4957f9bd8b61801846248013d71ebce not found: ID does not exist" Oct 30 18:57:39.063190 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.063168 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:57:39.074636 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.074614 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:57:39.140803 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:39.140780 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" path="/var/lib/kubelet/pods/116e8cd8-5f76-4b30-a446-0051cac4a571/volumes" Oct 30 18:57:43.333828 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:43.333793 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:57:47.255550 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:47.255514 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf1802_ffc1_4649_82a8_5a71b2827a01.slice/crio-9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec\": RecentStats: unable to find data in memory cache]" Oct 30 18:57:48.085386 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:48.085346 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf1802_ffc1_4649_82a8_5a71b2827a01.slice/crio-9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec\": RecentStats: unable to find data in memory cache]" Oct 30 18:57:49.737067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737034 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs"] Oct 30 18:57:49.737525 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737504 5014 cpu_manager.go:414] "containerMap: removing container" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="alertmanager" Oct 30 18:57:49.737628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737528 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="alertmanager" Oct 30 18:57:49.737628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737545 5014 cpu_manager.go:414] "containerMap: removing container" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="config-reloader" Oct 30 18:57:49.737628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737553 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="config-reloader" Oct 30 18:57:49.737628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737576 5014 cpu_manager.go:414] "containerMap: removing container" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="config-reloader" Oct 30 18:57:49.737628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737584 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="config-reloader" Oct 30 18:57:49.737628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737619 5014 cpu_manager.go:414] "containerMap: removing container" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="method-gate" Oct 30 18:57:49.737628 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737626 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="method-gate" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737642 5014 cpu_manager.go:414] "containerMap: removing container" podUID="27c1cc4d-7c04-40c6-8ed5-d421289676af" containerName="otc-container" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737649 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c1cc4d-7c04-40c6-8ed5-d421289676af" containerName="otc-container" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737660 5014 cpu_manager.go:414] "containerMap: removing container" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="prom-label-proxy" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737671 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="prom-label-proxy" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737681 5014 cpu_manager.go:414] "containerMap: removing container" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="thanos-sidecar" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737687 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="thanos-sidecar" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737700 5014 cpu_manager.go:414] "containerMap: removing container" podUID="f2e66b4d-3b28-4974-84e8-c1a31773f2c4" containerName="thanos-querier" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737706 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e66b4d-3b28-4974-84e8-c1a31773f2c4" containerName="thanos-querier" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737716 5014 cpu_manager.go:414] "containerMap: removing container" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="init-config-reloader" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737724 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="init-config-reloader" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737736 5014 cpu_manager.go:414] "containerMap: removing container" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="init-config-reloader" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737743 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="init-config-reloader" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737756 5014 cpu_manager.go:414] "containerMap: removing container" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="kube-rbac-proxy" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737762 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="kube-rbac-proxy" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737771 5014 cpu_manager.go:414] "containerMap: removing container" podUID="63ea83bc-7552-4cc1-805f-9f5eb6d10513" containerName="kube-rbac-proxy" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737778 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ea83bc-7552-4cc1-805f-9f5eb6d10513" containerName="kube-rbac-proxy" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737788 5014 cpu_manager.go:414] "containerMap: removing container" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="prometheus" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737795 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="prometheus" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737889 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c1cc4d-7c04-40c6-8ed5-d421289676af" containerName="otc-container" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737898 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="prom-label-proxy" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737905 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="thanos-sidecar" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737915 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e66b4d-3b28-4974-84e8-c1a31773f2c4" containerName="thanos-querier" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737925 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="kube-rbac-proxy" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737932 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ea83bc-7552-4cc1-805f-9f5eb6d10513" containerName="kube-rbac-proxy" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737939 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="prometheus" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737950 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="alertmanager" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737959 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="116e8cd8-5f76-4b30-a446-0051cac4a571" containerName="config-reloader" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737969 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="badf1802-ffc1-4649-82a8-5a71b2827a01" containerName="config-reloader" Oct 30 18:57:49.737979 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.737979 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2be25a-2bc2-4b8a-bfe1-0be5ed20428e" containerName="method-gate" Oct 30 18:57:49.743038 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.743019 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.769579 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.769547 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:57:49.769771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.769572 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:57:49.769771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.769620 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:57:49.769771 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.769641 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 18:57:49.769969 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.769918 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:57:49.772716 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.772697 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 18:57:49.772825 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.772771 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-b7254\"" Oct 30 18:57:49.774763 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.774742 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs"] Oct 30 18:57:49.782552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.782524 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z"] Oct 30 18:57:49.786655 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.786634 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:49.799575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.799555 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.799687 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.799617 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w48p\" (UniqueName: \"kubernetes.io/projected/841093e5-a2af-4af3-8635-d9bb3e914128-kube-api-access-6w48p\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.799735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.799710 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.799778 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.799748 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.799825 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.799809 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.804937 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.804913 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z"] Oct 30 18:57:49.861101 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.861065 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 18:57:49.861232 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.861189 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 18:57:49.861232 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.861227 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 18:57:49.861371 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.861269 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-hgvc8\"" Oct 30 18:57:49.900662 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.900636 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.900776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.900672 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82jbq\" (UniqueName: \"kubernetes.io/projected/09ee8f75-b086-4e90-acbc-01c763cce041-kube-api-access-82jbq\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:49.900776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.900698 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.900776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.900753 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:49.900970 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.900788 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/09ee8f75-b086-4e90-acbc-01c763cce041-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:49.900970 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.900852 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w48p\" (UniqueName: \"kubernetes.io/projected/841093e5-a2af-4af3-8635-d9bb3e914128-kube-api-access-6w48p\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.900970 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.900882 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.900970 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.900909 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:49.900970 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.900938 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:49.901272 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.901009 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.901272 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:49.901135 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 18:57:49.901272 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:49.901190 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-client-cert podName:841093e5-a2af-4af3-8635-d9bb3e914128 nodeName:}" failed. No retries permitted until 2026-10-30 18:57:50.401175822 +0000 UTC m=+3683.389516299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-zclqs" (UID: "841093e5-a2af-4af3-8635-d9bb3e914128") : secret "prometheus-operated-tls" not found Oct 30 18:57:49.901868 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.901841 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.903155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.903135 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.903249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.903198 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:49.926962 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:49.926938 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w48p\" (UniqueName: \"kubernetes.io/projected/841093e5-a2af-4af3-8635-d9bb3e914128-kube-api-access-6w48p\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:50.002161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.002111 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82jbq\" (UniqueName: \"kubernetes.io/projected/09ee8f75-b086-4e90-acbc-01c763cce041-kube-api-access-82jbq\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:50.002161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.002149 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:50.002290 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.002166 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/09ee8f75-b086-4e90-acbc-01c763cce041-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:50.002290 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.002200 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:50.002290 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.002218 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:50.002764 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.002747 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:50.002964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.002944 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:50.003021 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.002944 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:50.004446 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.004429 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/09ee8f75-b086-4e90-acbc-01c763cce041-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:50.030705 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.030683 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82jbq\" (UniqueName: \"kubernetes.io/projected/09ee8f75-b086-4e90-acbc-01c763cce041-kube-api-access-82jbq\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:50.106643 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.106623 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:57:50.405556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.405525 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:50.408056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.408030 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-zclqs\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:50.414501 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.414474 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z"] Oct 30 18:57:50.418933 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:57:50.418902 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ee8f75_b086_4e90_acbc_01c763cce041.slice/crio-dd3ef205b13f1c21f83c227a40604390ade19dd722e6ededbf2b22ab1d23cc5d WatchSource:0}: Error finding container dd3ef205b13f1c21f83c227a40604390ade19dd722e6ededbf2b22ab1d23cc5d: Status 404 returned error can't find the container with id dd3ef205b13f1c21f83c227a40604390ade19dd722e6ededbf2b22ab1d23cc5d Oct 30 18:57:50.656106 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.656057 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:50.954116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:50.954025 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs"] Oct 30 18:57:50.957825 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:57:50.957800 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841093e5_a2af_4af3_8635_d9bb3e914128.slice/crio-95ab64858079cf163ba522f3cb3db781c21a8a9fa0357698b65207baebaa6718 WatchSource:0}: Error finding container 95ab64858079cf163ba522f3cb3db781c21a8a9fa0357698b65207baebaa6718: Status 404 returned error can't find the container with id 95ab64858079cf163ba522f3cb3db781c21a8a9fa0357698b65207baebaa6718 Oct 30 18:57:51.075354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.075327 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" event={"ID":"841093e5-a2af-4af3-8635-d9bb3e914128","Type":"ContainerStarted","Data":"95ab64858079cf163ba522f3cb3db781c21a8a9fa0357698b65207baebaa6718"} Oct 30 18:57:51.077581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.077558 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" event={"ID":"09ee8f75-b086-4e90-acbc-01c763cce041","Type":"ContainerStarted","Data":"97a716ba4bd0b2b9d57bfce5f5f1ca6e11d0bbc5b778b92e725c7dd9e7047d3b"} Oct 30 18:57:51.077707 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.077635 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" event={"ID":"09ee8f75-b086-4e90-acbc-01c763cce041","Type":"ContainerStarted","Data":"4b0b3a79aa331b62fac478a30720dcd4ef84e5c2417757aed8b2d1b5b0feac37"} Oct 30 18:57:51.077707 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.077665 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" event={"ID":"09ee8f75-b086-4e90-acbc-01c763cce041","Type":"ContainerStarted","Data":"bf930d6005370b2a6603e02a80cd2a5c51d9a077ba1c72e7f62102d6e1deffc5"} Oct 30 18:57:51.077707 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.077688 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" event={"ID":"09ee8f75-b086-4e90-acbc-01c763cce041","Type":"ContainerStarted","Data":"dd3ef205b13f1c21f83c227a40604390ade19dd722e6ededbf2b22ab1d23cc5d"} Oct 30 18:57:51.119995 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.119945 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" podStartSLOduration=2.119931562 podStartE2EDuration="2.119931562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:51.119386844 +0000 UTC m=+3684.107727330" watchObservedRunningTime="2026-10-30 18:57:51.119931562 +0000 UTC m=+3684.108272049" Oct 30 18:57:51.327311 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.327252 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:51.332068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.332050 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.349014 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.348990 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:57:51.349135 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.349058 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:57:51.352909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.351645 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:57:51.352909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.352020 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:57:51.352909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.352058 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-lcs2t\"" Oct 30 18:57:51.352909 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.352114 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:57:51.354204 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.354182 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:51.414011 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.413984 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.414159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.414028 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e073df85-7c1b-40ea-b63c-a2f2069a2602-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.414159 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.414107 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.414288 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.414210 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.414288 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.414248 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbtqt\" (UniqueName: \"kubernetes.io/projected/e073df85-7c1b-40ea-b63c-a2f2069a2602-kube-api-access-hbtqt\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.414388 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.414284 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.514877 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.514838 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.515052 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.514930 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.515052 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.514977 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e073df85-7c1b-40ea-b63c-a2f2069a2602-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.515052 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.515037 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.515231 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.515085 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.515231 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.515115 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbtqt\" (UniqueName: \"kubernetes.io/projected/e073df85-7c1b-40ea-b63c-a2f2069a2602-kube-api-access-hbtqt\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.515231 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:51.515203 5014 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 30 18:57:51.515402 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:51.515273 5014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-otlp-tls-certs podName:e073df85-7c1b-40ea-b63c-a2f2069a2602 nodeName:}" failed. No retries permitted until 2026-10-30 18:57:52.015250898 +0000 UTC m=+3685.003591384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "e073df85-7c1b-40ea-b63c-a2f2069a2602") : secret "data-science-collector-collector-headless-tls" not found Oct 30 18:57:51.515984 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.515955 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e073df85-7c1b-40ea-b63c-a2f2069a2602-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.517704 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.517681 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.517807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.517681 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.517994 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.517979 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:51.538060 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:51.538042 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbtqt\" (UniqueName: \"kubernetes.io/projected/e073df85-7c1b-40ea-b63c-a2f2069a2602-kube-api-access-hbtqt\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:52.020131 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:52.020059 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:52.022805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:52.022776 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:52.083142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:52.083118 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" event={"ID":"841093e5-a2af-4af3-8635-d9bb3e914128","Type":"ContainerStarted","Data":"13ea1441eb9ede52887be0a04c820ca0d874868977098de9d3b206ab80f4c5bd"} Oct 30 18:57:52.083295 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:52.083278 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:52.123020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:52.122977 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" podStartSLOduration=3.122966111 podStartE2EDuration="3.122966111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:52.122764901 +0000 UTC m=+3685.111105390" watchObservedRunningTime="2026-10-30 18:57:52.122966111 +0000 UTC m=+3685.111306647" Oct 30 18:57:52.248156 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:52.248136 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:57:52.548379 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:52.548343 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:57:52.552457 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:57:52.552427 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode073df85_7c1b_40ea_b63c_a2f2069a2602.slice/crio-7878dd3043526e30e7136e1ab957a0301ef56ba753274e3f4d21938eccb28e77 WatchSource:0}: Error finding container 7878dd3043526e30e7136e1ab957a0301ef56ba753274e3f4d21938eccb28e77: Status 404 returned error can't find the container with id 7878dd3043526e30e7136e1ab957a0301ef56ba753274e3f4d21938eccb28e77 Oct 30 18:57:52.554673 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:52.554642 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:57:52.554769 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:52.554698 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:57:53.090260 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:53.090225 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e073df85-7c1b-40ea-b63c-a2f2069a2602","Type":"ContainerStarted","Data":"fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069"} Oct 30 18:57:53.090717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:53.090283 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e073df85-7c1b-40ea-b63c-a2f2069a2602","Type":"ContainerStarted","Data":"7878dd3043526e30e7136e1ab957a0301ef56ba753274e3f4d21938eccb28e77"} Oct 30 18:57:53.132573 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:53.132529 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.132516398 podStartE2EDuration="2.132516398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:53.131553636 +0000 UTC m=+3686.119894124" watchObservedRunningTime="2026-10-30 18:57:53.132516398 +0000 UTC m=+3686.120856884" Oct 30 18:57:58.093328 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:58.093292 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:57:58.139109 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:57:58.139079 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf1802_ffc1_4649_82a8_5a71b2827a01.slice/crio-9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec\": RecentStats: unable to find data in memory cache]" Oct 30 18:57:58.301135 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:58.301104 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv"] Oct 30 18:57:58.304704 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:58.304684 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" Oct 30 18:57:58.322186 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:58.322162 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-5mtd6\"" Oct 30 18:57:58.323860 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:58.323832 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv"] Oct 30 18:57:58.371422 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:58.371371 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5hdw\" (UniqueName: \"kubernetes.io/projected/ba31f50f-d878-472b-9986-03903b6472da-kube-api-access-p5hdw\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv\" (UID: \"ba31f50f-d878-472b-9986-03903b6472da\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" Oct 30 18:57:58.472642 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:58.472616 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5hdw\" (UniqueName: \"kubernetes.io/projected/ba31f50f-d878-472b-9986-03903b6472da-kube-api-access-p5hdw\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv\" (UID: \"ba31f50f-d878-472b-9986-03903b6472da\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" Oct 30 18:57:58.496190 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:58.496170 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5hdw\" (UniqueName: \"kubernetes.io/projected/ba31f50f-d878-472b-9986-03903b6472da-kube-api-access-p5hdw\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv\" (UID: \"ba31f50f-d878-472b-9986-03903b6472da\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" Oct 30 18:57:58.628688 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:58.628625 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" Oct 30 18:57:58.923196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:58.923126 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv"] Oct 30 18:57:58.927942 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:57:58.927915 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba31f50f_d878_472b_9986_03903b6472da.slice/crio-ddc4b827f7e1dbf86beaf9ec22969207fa4125016f3fc3782717874d3dfb7607 WatchSource:0}: Error finding container ddc4b827f7e1dbf86beaf9ec22969207fa4125016f3fc3782717874d3dfb7607: Status 404 returned error can't find the container with id ddc4b827f7e1dbf86beaf9ec22969207fa4125016f3fc3782717874d3dfb7607 Oct 30 18:57:59.121965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.121933 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" event={"ID":"ba31f50f-d878-472b-9986-03903b6472da","Type":"ContainerStarted","Data":"4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e"} Oct 30 18:57:59.122439 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.121983 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" event={"ID":"ba31f50f-d878-472b-9986-03903b6472da","Type":"ContainerStarted","Data":"ddc4b827f7e1dbf86beaf9ec22969207fa4125016f3fc3782717874d3dfb7607"} Oct 30 18:57:59.154105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.154059 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:57:59.158577 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.158560 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.160303 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.160254 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" podStartSLOduration=1.160238646 podStartE2EDuration="1.160238646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:59.159760884 +0000 UTC m=+3692.148101373" watchObservedRunningTime="2026-10-30 18:57:59.160238646 +0000 UTC m=+3692.148579134" Oct 30 18:57:59.182211 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.182128 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:57:59.224028 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.223985 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:57:59.228348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.228314 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.243933 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.243729 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:57:59.243933 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.243886 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:57:59.244205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.244015 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:57:59.244205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.244064 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:57:59.244205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.244096 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:57:59.244205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.244109 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:57:59.244205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.244128 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-qnsch\"" Oct 30 18:57:59.244205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.244173 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:57:59.244205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.244177 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:57:59.244205 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.244178 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:57:59.244575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.244220 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-j9qcn\"" Oct 30 18:57:59.256124 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.256101 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:57:59.280702 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.280674 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.280834 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.280716 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.280882 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.280842 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchpq\" (UniqueName: \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-kube-api-access-hchpq\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.280927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.280896 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.280965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.280930 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.281000 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.280962 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.281000 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.280994 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.281068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281037 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.281068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281063 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.281136 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281085 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.281170 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281139 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.281204 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281171 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.281242 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281201 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.281276 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281263 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3e48898-eb53-4585-9396-0fcd7ac596b2-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.281311 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281293 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.281348 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281317 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.281385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281350 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.281385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281370 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qppcq\" (UniqueName: \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-kube-api-access-qppcq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.281452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.281390 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.310688 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.310670 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:57:59.310688 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.310688 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:57:59.310927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.310913 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:57:59.382065 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382038 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382070 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382095 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3e48898-eb53-4585-9396-0fcd7ac596b2-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382119 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382151 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382184 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382211 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qppcq\" (UniqueName: \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-kube-api-access-qppcq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.382381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382234 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.382381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382278 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382302 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.382381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382334 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchpq\" (UniqueName: \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-kube-api-access-hchpq\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382381 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382357 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.382795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382387 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382412 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.382795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382446 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382488 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382517 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.382795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382542 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.382795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382626 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.382795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.382704 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.383223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.383022 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.385132 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.385105 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3e48898-eb53-4585-9396-0fcd7ac596b2-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.385243 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.385206 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.386046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.386017 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.386148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.386123 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.386225 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.386152 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.386622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.386288 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.386622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.386534 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.386622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.386558 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.386845 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.386726 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.387008 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.386980 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.387833 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.387812 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.388120 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.388099 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.388182 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.388147 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.388488 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.388465 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.388940 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.388924 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.404851 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.404830 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchpq\" (UniqueName: \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-kube-api-access-hchpq\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:57:59.404997 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.404979 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qppcq\" (UniqueName: \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-kube-api-access-qppcq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.477902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.477853 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:57:59.804284 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:57:59.804254 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:57:59.808057 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:57:59.808026 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940e125c_a268_4c06_87d0_6bbb63b03b1d.slice/crio-3fb5f9d91c3802af7d6840586b582ff41a3a9ec55fc353a980342cd620c56b46 WatchSource:0}: Error finding container 3fb5f9d91c3802af7d6840586b582ff41a3a9ec55fc353a980342cd620c56b46: Status 404 returned error can't find the container with id 3fb5f9d91c3802af7d6840586b582ff41a3a9ec55fc353a980342cd620c56b46 Oct 30 18:58:00.128137 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:00.128107 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"940e125c-a268-4c06-87d0-6bbb63b03b1d","Type":"ContainerStarted","Data":"1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b"} Oct 30 18:58:00.128480 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:00.128174 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"940e125c-a268-4c06-87d0-6bbb63b03b1d","Type":"ContainerStarted","Data":"3fb5f9d91c3802af7d6840586b582ff41a3a9ec55fc353a980342cd620c56b46"} Oct 30 18:58:02.215133 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:02.215104 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf1802_ffc1_4649_82a8_5a71b2827a01.slice/crio-9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec\": RecentStats: unable to find data in memory cache]" Oct 30 18:58:04.149794 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:04.149763 5014 generic.go:356] "Generic (PLEG): container finished" podID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerID="1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b" exitCode=0 Oct 30 18:58:04.150149 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:04.149838 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"940e125c-a268-4c06-87d0-6bbb63b03b1d","Type":"ContainerDied","Data":"1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b"} Oct 30 18:58:05.156587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:05.156555 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"940e125c-a268-4c06-87d0-6bbb63b03b1d","Type":"ContainerStarted","Data":"ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5"} Oct 30 18:58:05.156991 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:05.156632 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"940e125c-a268-4c06-87d0-6bbb63b03b1d","Type":"ContainerStarted","Data":"64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f"} Oct 30 18:58:05.156991 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:05.156925 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:58:05.229298 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:05.229251 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.6442372389999997 podStartE2EDuration="6.229240193s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.585002954s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:05.228248066 +0000 UTC m=+3698.216588553" watchObservedRunningTime="2026-10-30 18:58:05.229240193 +0000 UTC m=+3698.217580680" Oct 30 18:58:05.838971 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:05.838935 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:05.866878 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:05.866844 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:05.939378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:05.939347 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:05.939509 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:05.939487 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:05.953424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:05.953398 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:06.056831 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:06.056806 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/26c702426f1aeab2e9e6116e994cbd362b1230f690bb6d32002e36ee94fa3aec/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:06.083120 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:06.083101 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:06.149013 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:06.148958 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:06.487386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:06.487309 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:58:06.503489 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:58:06.503465 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e48898_eb53_4585_9396_0fcd7ac596b2.slice/crio-50741633f31aca9ab39138844d6b5a71e96699e3a6a58eae2427bb4240bc1df4 WatchSource:0}: Error finding container 50741633f31aca9ab39138844d6b5a71e96699e3a6a58eae2427bb4240bc1df4: Status 404 returned error can't find the container with id 50741633f31aca9ab39138844d6b5a71e96699e3a6a58eae2427bb4240bc1df4 Oct 30 18:58:07.167838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:07.167773 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b3e48898-eb53-4585-9396-0fcd7ac596b2","Type":"ContainerStarted","Data":"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d"} Oct 30 18:58:07.167838 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:07.167835 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b3e48898-eb53-4585-9396-0fcd7ac596b2","Type":"ContainerStarted","Data":"50741633f31aca9ab39138844d6b5a71e96699e3a6a58eae2427bb4240bc1df4"} Oct 30 18:58:08.150362 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:08.150335 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf1802_ffc1_4649_82a8_5a71b2827a01.slice/crio-9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec\": RecentStats: unable to find data in memory cache]" Oct 30 18:58:11.094874 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:11.094842 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf1802_ffc1_4649_82a8_5a71b2827a01.slice/crio-9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec\": RecentStats: unable to find data in memory cache]" Oct 30 18:58:11.095236 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:11.095156 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf1802_ffc1_4649_82a8_5a71b2827a01.slice/crio-9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec\": RecentStats: unable to find data in memory cache]" Oct 30 18:58:11.164084 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:11.164063 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:58:11.188040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:11.188010 5014 generic.go:356] "Generic (PLEG): container finished" podID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerID="be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d" exitCode=0 Oct 30 18:58:11.188152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:11.188087 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b3e48898-eb53-4585-9396-0fcd7ac596b2","Type":"ContainerDied","Data":"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d"} Oct 30 18:58:12.196105 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:12.196071 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b3e48898-eb53-4585-9396-0fcd7ac596b2","Type":"ContainerStarted","Data":"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608"} Oct 30 18:58:12.196482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:12.196132 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b3e48898-eb53-4585-9396-0fcd7ac596b2","Type":"ContainerStarted","Data":"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25"} Oct 30 18:58:12.196482 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:12.196159 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b3e48898-eb53-4585-9396-0fcd7ac596b2","Type":"ContainerStarted","Data":"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9"} Oct 30 18:58:12.249058 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:12.249017 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=9.863812365 podStartE2EDuration="13.249006638s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385194273s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:12.248123951 +0000 UTC m=+3705.236464455" watchObservedRunningTime="2026-10-30 18:58:12.249006638 +0000 UTC m=+3705.237347125" Oct 30 18:58:13.332346 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:13.332301 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:58:16.149305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:16.149265 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:17.261521 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:17.261493 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf1802_ffc1_4649_82a8_5a71b2827a01.slice/crio-9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec\": RecentStats: unable to find data in memory cache]" Oct 30 18:58:18.161986 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:18.161950 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf1802_ffc1_4649_82a8_5a71b2827a01.slice/crio-9fd2401fe04d222b795b5e5d1b94dee13d5a01a8cd708de007941f241e1779ec\": RecentStats: unable to find data in memory cache]" Oct 30 18:58:21.149345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:21.149309 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:21.154461 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:21.154433 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:21.244660 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:21.244641 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:26.230182 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:26.230114 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:26.230780 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:26.230393 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="e073df85-7c1b-40ea-b63c-a2f2069a2602" containerName="otc-container" containerID="cri-o://fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069" gracePeriod=30 Oct 30 18:58:28.587321 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.587297 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:28.628712 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.628661 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e073df85-7c1b-40ea-b63c-a2f2069a2602-kube-api-access-hbtqt\" (UniqueName: \"kubernetes.io/projected/e073df85-7c1b-40ea-b63c-a2f2069a2602-kube-api-access-hbtqt\") pod \"e073df85-7c1b-40ea-b63c-a2f2069a2602\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " Oct 30 18:58:28.628820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.628710 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e073df85-7c1b-40ea-b63c-a2f2069a2602-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e073df85-7c1b-40ea-b63c-a2f2069a2602-otc-internal\") pod \"e073df85-7c1b-40ea-b63c-a2f2069a2602\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " Oct 30 18:58:28.628820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.628784 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-tls-certs\") pod \"e073df85-7c1b-40ea-b63c-a2f2069a2602\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " Oct 30 18:58:28.628820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.628810 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-otlp-tls-certs\") pod \"e073df85-7c1b-40ea-b63c-a2f2069a2602\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " Oct 30 18:58:28.628965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.628939 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-monitoring-tls-certs\") pod \"e073df85-7c1b-40ea-b63c-a2f2069a2602\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " Oct 30 18:58:28.629055 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.629021 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-data-science-collector-ta-client-cert\") pod \"e073df85-7c1b-40ea-b63c-a2f2069a2602\" (UID: \"e073df85-7c1b-40ea-b63c-a2f2069a2602\") " Oct 30 18:58:28.629152 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.629125 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e073df85-7c1b-40ea-b63c-a2f2069a2602-otc-internal" pod "e073df85-7c1b-40ea-b63c-a2f2069a2602" (UID: "e073df85-7c1b-40ea-b63c-a2f2069a2602"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:28.629385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.629357 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e073df85-7c1b-40ea-b63c-a2f2069a2602-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:28.633025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.631668 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-tls-certs" pod "e073df85-7c1b-40ea-b63c-a2f2069a2602" (UID: "e073df85-7c1b-40ea-b63c-a2f2069a2602"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:28.633025 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.631755 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-otlp-tls-certs" pod "e073df85-7c1b-40ea-b63c-a2f2069a2602" (UID: "e073df85-7c1b-40ea-b63c-a2f2069a2602"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:28.633309 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.633083 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e073df85-7c1b-40ea-b63c-a2f2069a2602-kube-api-access-hbtqt" pod "e073df85-7c1b-40ea-b63c-a2f2069a2602" (UID: "e073df85-7c1b-40ea-b63c-a2f2069a2602"). InnerVolumeSpecName "kube-api-access-hbtqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:28.633309 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.633220 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-data-science-collector-ta-client-cert" pod "e073df85-7c1b-40ea-b63c-a2f2069a2602" (UID: "e073df85-7c1b-40ea-b63c-a2f2069a2602"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:28.633451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.633434 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-monitoring-tls-certs" pod "e073df85-7c1b-40ea-b63c-a2f2069a2602" (UID: "e073df85-7c1b-40ea-b63c-a2f2069a2602"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:28.729837 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.729812 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:28.729837 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.729833 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:28.729977 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.729842 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:28.729977 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.729852 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e073df85-7c1b-40ea-b63c-a2f2069a2602-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:28.729977 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:28.729862 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbtqt\" (UniqueName: \"kubernetes.io/projected/e073df85-7c1b-40ea-b63c-a2f2069a2602-kube-api-access-hbtqt\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:29.279113 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.279075 5014 generic.go:356] "Generic (PLEG): container finished" podID="e073df85-7c1b-40ea-b63c-a2f2069a2602" containerID="fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069" exitCode=0 Oct 30 18:58:29.279263 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.279134 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.279263 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.279158 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e073df85-7c1b-40ea-b63c-a2f2069a2602","Type":"ContainerDied","Data":"fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069"} Oct 30 18:58:29.279263 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.279231 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e073df85-7c1b-40ea-b63c-a2f2069a2602","Type":"ContainerDied","Data":"7878dd3043526e30e7136e1ab957a0301ef56ba753274e3f4d21938eccb28e77"} Oct 30 18:58:29.279263 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.279254 5014 scope.go:122] "RemoveContainer" containerID="fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069" Oct 30 18:58:29.288849 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.288828 5014 scope.go:122] "RemoveContainer" containerID="fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069" Oct 30 18:58:29.289185 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:29.289167 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069\": container with ID starting with fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069 not found: ID does not exist" containerID="fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069" Oct 30 18:58:29.289229 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.289193 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069"} err="failed to get container status \"fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069\": rpc error: code = NotFound desc = could not find container \"fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069\": container with ID starting with fb4eefc885482e41c87911e9d2696dd3b05e79f15b4184a1ab948e2694413069 not found: ID does not exist" Oct 30 18:58:29.330168 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.330141 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:29.342085 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.342066 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:29.371476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.371447 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:29.371961 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.371943 5014 cpu_manager.go:414] "containerMap: removing container" podUID="e073df85-7c1b-40ea-b63c-a2f2069a2602" containerName="otc-container" Oct 30 18:58:29.372060 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.371963 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e073df85-7c1b-40ea-b63c-a2f2069a2602" containerName="otc-container" Oct 30 18:58:29.372129 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.372102 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e073df85-7c1b-40ea-b63c-a2f2069a2602" containerName="otc-container" Oct 30 18:58:29.376531 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.376513 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.391236 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.391214 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:58:29.391346 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.391321 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-lcs2t\"" Oct 30 18:58:29.391426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.391381 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:58:29.391426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.391406 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:58:29.391426 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.391410 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:58:29.391587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.391428 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:58:29.399627 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.399567 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:29.436126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.436101 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.436249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.436135 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpwkh\" (UniqueName: \"kubernetes.io/projected/316c1947-7547-4666-9f3d-9d177fa7adb3-kube-api-access-zpwkh\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.436249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.436161 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/316c1947-7547-4666-9f3d-9d177fa7adb3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.436347 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.436288 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.436386 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.436347 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.436506 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.436487 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.537094 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.537066 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.537196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.537130 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.537196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.537156 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.537196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.537172 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpwkh\" (UniqueName: \"kubernetes.io/projected/316c1947-7547-4666-9f3d-9d177fa7adb3-kube-api-access-zpwkh\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.537196 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.537189 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/316c1947-7547-4666-9f3d-9d177fa7adb3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.537399 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.537208 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.537928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.537908 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/316c1947-7547-4666-9f3d-9d177fa7adb3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.539937 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.539911 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.540063 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.539917 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.540169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.540147 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.540315 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.540297 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.559768 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.559750 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpwkh\" (UniqueName: \"kubernetes.io/projected/316c1947-7547-4666-9f3d-9d177fa7adb3-kube-api-access-zpwkh\") pod \"data-science-collector-collector-0\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.689135 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.689087 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:29.985619 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.985527 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:29.989709 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:58:29.989668 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316c1947_7547_4666_9f3d_9d177fa7adb3.slice/crio-4a6374f92428d5c32db48e5646855807a44c36de0d6e5ea36660a041fc6bdbb1 WatchSource:0}: Error finding container 4a6374f92428d5c32db48e5646855807a44c36de0d6e5ea36660a041fc6bdbb1: Status 404 returned error can't find the container with id 4a6374f92428d5c32db48e5646855807a44c36de0d6e5ea36660a041fc6bdbb1 Oct 30 18:58:29.992001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.991970 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:58:29.992087 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:29.992033 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:58:30.285990 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:30.285908 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"316c1947-7547-4666-9f3d-9d177fa7adb3","Type":"ContainerStarted","Data":"ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d"} Oct 30 18:58:30.285990 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:30.285968 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"316c1947-7547-4666-9f3d-9d177fa7adb3","Type":"ContainerStarted","Data":"4a6374f92428d5c32db48e5646855807a44c36de0d6e5ea36660a041fc6bdbb1"} Oct 30 18:58:30.327930 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:30.327886 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.327872063 podStartE2EDuration="1.327872063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:30.327351662 +0000 UTC m=+3723.315692148" watchObservedRunningTime="2026-10-30 18:58:30.327872063 +0000 UTC m=+3723.316212549" Oct 30 18:58:31.142157 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:31.142123 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e073df85-7c1b-40ea-b63c-a2f2069a2602" path="/var/lib/kubelet/pods/e073df85-7c1b-40ea-b63c-a2f2069a2602/volumes" Oct 30 18:58:41.313286 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:41.313252 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:41.313702 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:41.313460 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="316c1947-7547-4666-9f3d-9d177fa7adb3" containerName="otc-container" containerID="cri-o://ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d" gracePeriod=30 Oct 30 18:58:43.336448 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:43.336399 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:58:43.662679 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.662650 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:43.756035 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.756004 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/316c1947-7547-4666-9f3d-9d177fa7adb3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/316c1947-7547-4666-9f3d-9d177fa7adb3-otc-internal\") pod \"316c1947-7547-4666-9f3d-9d177fa7adb3\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " Oct 30 18:58:43.756167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.756072 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-data-science-collector-ta-client-cert\") pod \"316c1947-7547-4666-9f3d-9d177fa7adb3\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " Oct 30 18:58:43.756167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.756092 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-monitoring-tls-certs\") pod \"316c1947-7547-4666-9f3d-9d177fa7adb3\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " Oct 30 18:58:43.756167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.756107 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-otlp-tls-certs\") pod \"316c1947-7547-4666-9f3d-9d177fa7adb3\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " Oct 30 18:58:43.756167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.756135 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-tls-certs\") pod \"316c1947-7547-4666-9f3d-9d177fa7adb3\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " Oct 30 18:58:43.756167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.756162 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316c1947-7547-4666-9f3d-9d177fa7adb3-kube-api-access-zpwkh\" (UniqueName: \"kubernetes.io/projected/316c1947-7547-4666-9f3d-9d177fa7adb3-kube-api-access-zpwkh\") pod \"316c1947-7547-4666-9f3d-9d177fa7adb3\" (UID: \"316c1947-7547-4666-9f3d-9d177fa7adb3\") " Oct 30 18:58:43.756478 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.756430 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316c1947-7547-4666-9f3d-9d177fa7adb3-otc-internal" pod "316c1947-7547-4666-9f3d-9d177fa7adb3" (UID: "316c1947-7547-4666-9f3d-9d177fa7adb3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:43.758627 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.758578 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-tls-certs" pod "316c1947-7547-4666-9f3d-9d177fa7adb3" (UID: "316c1947-7547-4666-9f3d-9d177fa7adb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:43.758943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.758920 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-data-science-collector-ta-client-cert" pod "316c1947-7547-4666-9f3d-9d177fa7adb3" (UID: "316c1947-7547-4666-9f3d-9d177fa7adb3"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:43.758943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.758936 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-monitoring-tls-certs" pod "316c1947-7547-4666-9f3d-9d177fa7adb3" (UID: "316c1947-7547-4666-9f3d-9d177fa7adb3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:43.759082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.759020 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316c1947-7547-4666-9f3d-9d177fa7adb3-kube-api-access-zpwkh" pod "316c1947-7547-4666-9f3d-9d177fa7adb3" (UID: "316c1947-7547-4666-9f3d-9d177fa7adb3"). InnerVolumeSpecName "kube-api-access-zpwkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:43.759082 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.759069 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-otlp-tls-certs" pod "316c1947-7547-4666-9f3d-9d177fa7adb3" (UID: "316c1947-7547-4666-9f3d-9d177fa7adb3"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:43.857570 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.857546 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/316c1947-7547-4666-9f3d-9d177fa7adb3-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:43.857691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.857571 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:43.857691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.857587 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:43.857691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.857613 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:43.857691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.857622 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/316c1947-7547-4666-9f3d-9d177fa7adb3-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:43.857691 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:43.857630 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpwkh\" (UniqueName: \"kubernetes.io/projected/316c1947-7547-4666-9f3d-9d177fa7adb3-kube-api-access-zpwkh\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:44.357940 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.357911 5014 generic.go:356] "Generic (PLEG): container finished" podID="316c1947-7547-4666-9f3d-9d177fa7adb3" containerID="ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d" exitCode=0 Oct 30 18:58:44.358341 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.357960 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.358341 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.358008 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"316c1947-7547-4666-9f3d-9d177fa7adb3","Type":"ContainerDied","Data":"ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d"} Oct 30 18:58:44.358341 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.358056 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"316c1947-7547-4666-9f3d-9d177fa7adb3","Type":"ContainerDied","Data":"4a6374f92428d5c32db48e5646855807a44c36de0d6e5ea36660a041fc6bdbb1"} Oct 30 18:58:44.358341 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.358080 5014 scope.go:122] "RemoveContainer" containerID="ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d" Oct 30 18:58:44.368433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.368415 5014 scope.go:122] "RemoveContainer" containerID="ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d" Oct 30 18:58:44.368736 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:44.368713 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d\": container with ID starting with ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d not found: ID does not exist" containerID="ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d" Oct 30 18:58:44.368801 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.368745 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d"} err="failed to get container status \"ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d\": rpc error: code = NotFound desc = could not find container \"ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d\": container with ID starting with ab5de3ffe89d2a7087e6884e57cdfdccfb3b11c7e2b1735b386c04b54d7bab9d not found: ID does not exist" Oct 30 18:58:44.408280 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.408255 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:44.419729 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.419705 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:44.452847 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.452828 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:44.453305 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.453288 5014 cpu_manager.go:414] "containerMap: removing container" podUID="316c1947-7547-4666-9f3d-9d177fa7adb3" containerName="otc-container" Oct 30 18:58:44.453366 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.453306 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316c1947-7547-4666-9f3d-9d177fa7adb3" containerName="otc-container" Oct 30 18:58:44.453412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.453381 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="316c1947-7547-4666-9f3d-9d177fa7adb3" containerName="otc-container" Oct 30 18:58:44.457887 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.457871 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.472532 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.472513 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:58:44.472711 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.472683 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-lcs2t\"" Oct 30 18:58:44.472818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.472738 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:58:44.472818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.472751 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:58:44.472818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.472755 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:58:44.472818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.472780 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:58:44.478464 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.478433 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:44.564069 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.564042 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.564178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.564079 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.564178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.564108 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnt76\" (UniqueName: \"kubernetes.io/projected/4882ef1c-74d9-4a5e-9481-32461fd011f7-kube-api-access-dnt76\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.564178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.564168 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4882ef1c-74d9-4a5e-9481-32461fd011f7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.564378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.564255 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.564378 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.564276 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.665437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.665372 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnt76\" (UniqueName: \"kubernetes.io/projected/4882ef1c-74d9-4a5e-9481-32461fd011f7-kube-api-access-dnt76\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.665437 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.665415 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4882ef1c-74d9-4a5e-9481-32461fd011f7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.665682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.665444 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.665682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.665465 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.665682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.665510 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.665682 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.665534 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.666253 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.666224 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4882ef1c-74d9-4a5e-9481-32461fd011f7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.668098 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.668079 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.668191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.668157 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.668191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.668168 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.668281 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.668195 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.687896 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.687873 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnt76\" (UniqueName: \"kubernetes.io/projected/4882ef1c-74d9-4a5e-9481-32461fd011f7-kube-api-access-dnt76\") pod \"data-science-collector-collector-0\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:44.769781 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:44.769760 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:45.068351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:45.068314 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:45.072297 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:58:45.072273 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4882ef1c_74d9_4a5e_9481_32461fd011f7.slice/crio-993df3585b649afc63d9418a8bd34c6e2c304ce62c4836744f973351d4c05a76 WatchSource:0}: Error finding container 993df3585b649afc63d9418a8bd34c6e2c304ce62c4836744f973351d4c05a76: Status 404 returned error can't find the container with id 993df3585b649afc63d9418a8bd34c6e2c304ce62c4836744f973351d4c05a76 Oct 30 18:58:45.074433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:45.074402 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:58:45.074523 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:45.074461 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:58:45.141435 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:45.141406 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="316c1947-7547-4666-9f3d-9d177fa7adb3" path="/var/lib/kubelet/pods/316c1947-7547-4666-9f3d-9d177fa7adb3/volumes" Oct 30 18:58:45.365932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:45.365856 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4882ef1c-74d9-4a5e-9481-32461fd011f7","Type":"ContainerStarted","Data":"bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc"} Oct 30 18:58:45.365932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:45.365905 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4882ef1c-74d9-4a5e-9481-32461fd011f7","Type":"ContainerStarted","Data":"993df3585b649afc63d9418a8bd34c6e2c304ce62c4836744f973351d4c05a76"} Oct 30 18:58:45.407869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:45.407813 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.407797007 podStartE2EDuration="1.407797007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:45.406839577 +0000 UTC m=+3738.395180071" watchObservedRunningTime="2026-10-30 18:58:45.407797007 +0000 UTC m=+3738.396137496" Oct 30 18:58:48.202161 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.202080 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs"] Oct 30 18:58:48.202809 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.202300 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" podUID="841093e5-a2af-4af3-8635-d9bb3e914128" containerName="kube-rbac-proxy" containerID="cri-o://13ea1441eb9ede52887be0a04c820ca0d874868977098de9d3b206ab80f4c5bd" gracePeriod=30 Oct 30 18:58:48.214010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.213987 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z"] Oct 30 18:58:48.214217 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.214196 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="kube-rbac-proxy" containerID="cri-o://bf930d6005370b2a6603e02a80cd2a5c51d9a077ba1c72e7f62102d6e1deffc5" gracePeriod=30 Oct 30 18:58:48.214282 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.214197 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="prom-label-proxy" containerID="cri-o://97a716ba4bd0b2b9d57bfce5f5f1ca6e11d0bbc5b778b92e725c7dd9e7047d3b" gracePeriod=30 Oct 30 18:58:48.214282 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.214221 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="method-gate" containerID="cri-o://4b0b3a79aa331b62fac478a30720dcd4ef84e5c2417757aed8b2d1b5b0feac37" gracePeriod=30 Oct 30 18:58:48.383383 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.383227 5014 generic.go:356] "Generic (PLEG): container finished" podID="841093e5-a2af-4af3-8635-d9bb3e914128" containerID="13ea1441eb9ede52887be0a04c820ca0d874868977098de9d3b206ab80f4c5bd" exitCode=0 Oct 30 18:58:48.383383 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.383294 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" event={"ID":"841093e5-a2af-4af3-8635-d9bb3e914128","Type":"ContainerDied","Data":"13ea1441eb9ede52887be0a04c820ca0d874868977098de9d3b206ab80f4c5bd"} Oct 30 18:58:48.385956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.385706 5014 generic.go:356] "Generic (PLEG): container finished" podID="09ee8f75-b086-4e90-acbc-01c763cce041" containerID="97a716ba4bd0b2b9d57bfce5f5f1ca6e11d0bbc5b778b92e725c7dd9e7047d3b" exitCode=0 Oct 30 18:58:48.385956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.385731 5014 generic.go:356] "Generic (PLEG): container finished" podID="09ee8f75-b086-4e90-acbc-01c763cce041" containerID="4b0b3a79aa331b62fac478a30720dcd4ef84e5c2417757aed8b2d1b5b0feac37" exitCode=0 Oct 30 18:58:48.385956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.385739 5014 generic.go:356] "Generic (PLEG): container finished" podID="09ee8f75-b086-4e90-acbc-01c763cce041" containerID="bf930d6005370b2a6603e02a80cd2a5c51d9a077ba1c72e7f62102d6e1deffc5" exitCode=0 Oct 30 18:58:48.385956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.385808 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" event={"ID":"09ee8f75-b086-4e90-acbc-01c763cce041","Type":"ContainerDied","Data":"97a716ba4bd0b2b9d57bfce5f5f1ca6e11d0bbc5b778b92e725c7dd9e7047d3b"} Oct 30 18:58:48.385956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.385860 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" event={"ID":"09ee8f75-b086-4e90-acbc-01c763cce041","Type":"ContainerDied","Data":"4b0b3a79aa331b62fac478a30720dcd4ef84e5c2417757aed8b2d1b5b0feac37"} Oct 30 18:58:48.385956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.385899 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" event={"ID":"09ee8f75-b086-4e90-acbc-01c763cce041","Type":"ContainerDied","Data":"bf930d6005370b2a6603e02a80cd2a5c51d9a077ba1c72e7f62102d6e1deffc5"} Oct 30 18:58:48.467220 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.467158 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:58:48.504735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.504707 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:58:48.526077 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.525732 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv"] Oct 30 18:58:48.526077 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.525931 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" podUID="ba31f50f-d878-472b-9986-03903b6472da" containerName="thanos-querier" containerID="cri-o://4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e" gracePeriod=30 Oct 30 18:58:48.598649 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.598615 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-ca\") pod \"841093e5-a2af-4af3-8635-d9bb3e914128\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " Oct 30 18:58:48.598805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.598669 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-config\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-config\") pod \"09ee8f75-b086-4e90-acbc-01c763cce041\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " Oct 30 18:58:48.598805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.598711 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-client-cert\") pod \"841093e5-a2af-4af3-8635-d9bb3e914128\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " Oct 30 18:58:48.598805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.598755 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841093e5-a2af-4af3-8635-d9bb3e914128-kube-api-access-6w48p\" (UniqueName: \"kubernetes.io/projected/841093e5-a2af-4af3-8635-d9bb3e914128-kube-api-access-6w48p\") pod \"841093e5-a2af-4af3-8635-d9bb3e914128\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " Oct 30 18:58:48.598805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.598777 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-tls\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-tls\") pod \"841093e5-a2af-4af3-8635-d9bb3e914128\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " Oct 30 18:58:48.598953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.598806 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-kube-rbac-proxy-config\") pod \"841093e5-a2af-4af3-8635-d9bb3e914128\" (UID: \"841093e5-a2af-4af3-8635-d9bb3e914128\") " Oct 30 18:58:48.598953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.598852 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ee8f75-b086-4e90-acbc-01c763cce041-tls\" (UniqueName: \"kubernetes.io/secret/09ee8f75-b086-4e90-acbc-01c763cce041-tls\") pod \"09ee8f75-b086-4e90-acbc-01c763cce041\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " Oct 30 18:58:48.598953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.598888 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-prometheus-tls-ca\") pod \"09ee8f75-b086-4e90-acbc-01c763cce041\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " Oct 30 18:58:48.598953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.598909 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ee8f75-b086-4e90-acbc-01c763cce041-kube-api-access-82jbq\" (UniqueName: \"kubernetes.io/projected/09ee8f75-b086-4e90-acbc-01c763cce041-kube-api-access-82jbq\") pod \"09ee8f75-b086-4e90-acbc-01c763cce041\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " Oct 30 18:58:48.598953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.598934 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-client-ca\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-client-ca\") pod \"09ee8f75-b086-4e90-acbc-01c763cce041\" (UID: \"09ee8f75-b086-4e90-acbc-01c763cce041\") " Oct 30 18:58:48.599354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.599335 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-client-ca" pod "09ee8f75-b086-4e90-acbc-01c763cce041" (UID: "09ee8f75-b086-4e90-acbc-01c763cce041"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:48.599569 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.599556 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-ca" pod "841093e5-a2af-4af3-8635-d9bb3e914128" (UID: "841093e5-a2af-4af3-8635-d9bb3e914128"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:48.600175 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.600152 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-config" pod "09ee8f75-b086-4e90-acbc-01c763cce041" (UID: "09ee8f75-b086-4e90-acbc-01c763cce041"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:48.600730 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.600678 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-prometheus-tls-ca" pod "09ee8f75-b086-4e90-acbc-01c763cce041" (UID: "09ee8f75-b086-4e90-acbc-01c763cce041"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:48.602665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.602361 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-kube-rbac-proxy-config" pod "841093e5-a2af-4af3-8635-d9bb3e914128" (UID: "841093e5-a2af-4af3-8635-d9bb3e914128"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:48.602665 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.602568 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841093e5-a2af-4af3-8635-d9bb3e914128-kube-api-access-6w48p" pod "841093e5-a2af-4af3-8635-d9bb3e914128" (UID: "841093e5-a2af-4af3-8635-d9bb3e914128"). InnerVolumeSpecName "kube-api-access-6w48p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:48.604017 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.603992 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ee8f75-b086-4e90-acbc-01c763cce041-tls" pod "09ee8f75-b086-4e90-acbc-01c763cce041" (UID: "09ee8f75-b086-4e90-acbc-01c763cce041"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:48.604389 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.604369 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ee8f75-b086-4e90-acbc-01c763cce041-kube-api-access-82jbq" pod "09ee8f75-b086-4e90-acbc-01c763cce041" (UID: "09ee8f75-b086-4e90-acbc-01c763cce041"). InnerVolumeSpecName "kube-api-access-82jbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:48.605071 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.605050 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-client-cert" pod "841093e5-a2af-4af3-8635-d9bb3e914128" (UID: "841093e5-a2af-4af3-8635-d9bb3e914128"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:48.605158 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.605065 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-tls" pod "841093e5-a2af-4af3-8635-d9bb3e914128" (UID: "841093e5-a2af-4af3-8635-d9bb3e914128"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:48.690247 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.689711 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:58:48.690247 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.689930 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="alertmanager" containerID="cri-o://64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f" gracePeriod=120 Oct 30 18:58:48.690247 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.690071 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="config-reloader" containerID="cri-o://ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5" gracePeriod=120 Oct 30 18:58:48.692167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.692120 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:58:48.696277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.692451 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="prometheus" containerID="cri-o://779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9" gracePeriod=600 Oct 30 18:58:48.696277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.692664 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="thanos-sidecar" containerID="cri-o://2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608" gracePeriod=600 Oct 30 18:58:48.696277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.694024 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="config-reloader" containerID="cri-o://babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25" gracePeriod=600 Oct 30 18:58:48.701007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.700271 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-prometheus-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.701007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.700298 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82jbq\" (UniqueName: \"kubernetes.io/projected/09ee8f75-b086-4e90-acbc-01c763cce041-kube-api-access-82jbq\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.701007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.700310 5014 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-client-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.701007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.700328 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.701007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.700338 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09ee8f75-b086-4e90-acbc-01c763cce041-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.701007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.700345 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-prometheus-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.701007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.700354 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w48p\" (UniqueName: \"kubernetes.io/projected/841093e5-a2af-4af3-8635-d9bb3e914128-kube-api-access-6w48p\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.701007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.700362 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.701007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.700375 5014 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/841093e5-a2af-4af3-8635-d9bb3e914128-kube-rbac-proxy-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.701007 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.700386 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/09ee8f75-b086-4e90-acbc-01c763cce041-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.835191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:48.835168 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" Oct 30 18:58:49.002969 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.002942 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba31f50f-d878-472b-9986-03903b6472da-kube-api-access-p5hdw\" (UniqueName: \"kubernetes.io/projected/ba31f50f-d878-472b-9986-03903b6472da-kube-api-access-p5hdw\") pod \"ba31f50f-d878-472b-9986-03903b6472da\" (UID: \"ba31f50f-d878-472b-9986-03903b6472da\") " Oct 30 18:58:49.005360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.005336 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba31f50f-d878-472b-9986-03903b6472da-kube-api-access-p5hdw" pod "ba31f50f-d878-472b-9986-03903b6472da" (UID: "ba31f50f-d878-472b-9986-03903b6472da"). InnerVolumeSpecName "kube-api-access-p5hdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:49.028975 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.028930 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:49.104360 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104332 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-secret-prometheus-web-tls-ca\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.104497 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104371 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-tls-assets\" (UniqueName: \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-tls-assets\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.104497 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104484 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.104660 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104584 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.104660 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104637 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.104794 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104686 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-config\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-config\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.104794 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104729 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.104794 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104764 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e48898-eb53-4585-9396-0fcd7ac596b2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3e48898-eb53-4585-9396-0fcd7ac596b2-config-out\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.104964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104813 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.104964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104846 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-thanos-prometheus-http-client-file\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.104964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104891 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.104964 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.104933 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-kube-api-access-hchpq\" (UniqueName: \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-kube-api-access-hchpq\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.105190 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.105015 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config\") pod \"b3e48898-eb53-4585-9396-0fcd7ac596b2\" (UID: \"b3e48898-eb53-4585-9396-0fcd7ac596b2\") " Oct 30 18:58:49.105620 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.105426 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5hdw\" (UniqueName: \"kubernetes.io/projected/ba31f50f-d878-472b-9986-03903b6472da-kube-api-access-p5hdw\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.105935 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.105909 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-2" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:49.106033 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.105708 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-0" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:49.107020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.106984 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-1" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:49.107392 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.107367 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-tls-assets" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:49.107820 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.107791 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-secret-prometheus-web-tls-ca" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.109975 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.109947 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-thanos-prometheus-http-client-file" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.110151 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.110123 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.110903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.110871 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-kube-api-access-hchpq" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "kube-api-access-hchpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:49.111115 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.111087 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.111331 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.111297 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-config" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.111433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.111340 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e48898-eb53-4585-9396-0fcd7ac596b2-config-out" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:49.112223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.112203 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "pvc-c02e8472-bb1a-4974-aa89-86eef715889b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:58:49.124056 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.124026 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config" pod "b3e48898-eb53-4585-9396-0fcd7ac596b2" (UID: "b3e48898-eb53-4585-9396-0fcd7ac596b2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.205850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205827 5014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.205850 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205848 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.206180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205859 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.206180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205888 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:58:49.206180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205902 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.206180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205911 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.206180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205920 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.206180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205928 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3e48898-eb53-4585-9396-0fcd7ac596b2-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.206180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205937 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.206180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205950 5014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.206180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205964 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b3e48898-eb53-4585-9396-0fcd7ac596b2-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.206180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205977 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hchpq\" (UniqueName: \"kubernetes.io/projected/b3e48898-eb53-4585-9396-0fcd7ac596b2-kube-api-access-hchpq\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.206180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.205990 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3e48898-eb53-4585-9396-0fcd7ac596b2-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.217193 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.217142 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c02e8472-bb1a-4974-aa89-86eef715889b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:58:49.307358 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.307334 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" Oct 30 18:58:49.323710 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.323686 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:49.323930 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.323906 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="4882ef1c-74d9-4a5e-9481-32461fd011f7" containerName="otc-container" containerID="cri-o://bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc" gracePeriod=30 Oct 30 18:58:49.394806 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.394769 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" event={"ID":"09ee8f75-b086-4e90-acbc-01c763cce041","Type":"ContainerDied","Data":"dd3ef205b13f1c21f83c227a40604390ade19dd722e6ededbf2b22ab1d23cc5d"} Oct 30 18:58:49.394806 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.394796 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z" Oct 30 18:58:49.395075 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.394864 5014 scope.go:122] "RemoveContainer" containerID="97a716ba4bd0b2b9d57bfce5f5f1ca6e11d0bbc5b778b92e725c7dd9e7047d3b" Oct 30 18:58:49.396556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.396534 5014 generic.go:356] "Generic (PLEG): container finished" podID="ba31f50f-d878-472b-9986-03903b6472da" containerID="4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e" exitCode=0 Oct 30 18:58:49.396653 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.396575 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" Oct 30 18:58:49.396713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.396641 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" event={"ID":"ba31f50f-d878-472b-9986-03903b6472da","Type":"ContainerDied","Data":"4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e"} Oct 30 18:58:49.396713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.396677 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv" event={"ID":"ba31f50f-d878-472b-9986-03903b6472da","Type":"ContainerDied","Data":"ddc4b827f7e1dbf86beaf9ec22969207fa4125016f3fc3782717874d3dfb7607"} Oct 30 18:58:49.399324 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.399302 5014 generic.go:356] "Generic (PLEG): container finished" podID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerID="ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5" exitCode=0 Oct 30 18:58:49.399421 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.399367 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"940e125c-a268-4c06-87d0-6bbb63b03b1d","Type":"ContainerDied","Data":"ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5"} Oct 30 18:58:49.401815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.401793 5014 generic.go:356] "Generic (PLEG): container finished" podID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerID="2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608" exitCode=0 Oct 30 18:58:49.401815 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.401816 5014 generic.go:356] "Generic (PLEG): container finished" podID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerID="babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25" exitCode=0 Oct 30 18:58:49.401966 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.401825 5014 generic.go:356] "Generic (PLEG): container finished" podID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerID="779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9" exitCode=0 Oct 30 18:58:49.401966 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.401870 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:58:49.401966 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.401875 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b3e48898-eb53-4585-9396-0fcd7ac596b2","Type":"ContainerDied","Data":"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608"} Oct 30 18:58:49.401966 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.401937 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b3e48898-eb53-4585-9396-0fcd7ac596b2","Type":"ContainerDied","Data":"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25"} Oct 30 18:58:49.402148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.401977 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b3e48898-eb53-4585-9396-0fcd7ac596b2","Type":"ContainerDied","Data":"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9"} Oct 30 18:58:49.402148 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.402015 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b3e48898-eb53-4585-9396-0fcd7ac596b2","Type":"ContainerDied","Data":"50741633f31aca9ab39138844d6b5a71e96699e3a6a58eae2427bb4240bc1df4"} Oct 30 18:58:49.403233 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.403215 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" event={"ID":"841093e5-a2af-4af3-8635-d9bb3e914128","Type":"ContainerDied","Data":"95ab64858079cf163ba522f3cb3db781c21a8a9fa0357698b65207baebaa6718"} Oct 30 18:58:49.403325 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.403260 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs" Oct 30 18:58:49.453293 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.453266 5014 scope.go:122] "RemoveContainer" containerID="4b0b3a79aa331b62fac478a30720dcd4ef84e5c2417757aed8b2d1b5b0feac37" Oct 30 18:58:49.461427 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.461407 5014 scope.go:122] "RemoveContainer" containerID="bf930d6005370b2a6603e02a80cd2a5c51d9a077ba1c72e7f62102d6e1deffc5" Oct 30 18:58:49.469138 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.469118 5014 scope.go:122] "RemoveContainer" containerID="4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e" Oct 30 18:58:49.477310 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.477290 5014 scope.go:122] "RemoveContainer" containerID="4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e" Oct 30 18:58:49.477587 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:49.477564 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e\": container with ID starting with 4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e not found: ID does not exist" containerID="4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e" Oct 30 18:58:49.477713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.477623 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e"} err="failed to get container status \"4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e\": rpc error: code = NotFound desc = could not find container \"4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e\": container with ID starting with 4e5cb4fecf1dcdc4f3619be8af6a6911c7a473f03d3dcebc743eb2caac4b817e not found: ID does not exist" Oct 30 18:58:49.477713 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.477652 5014 scope.go:122] "RemoveContainer" containerID="2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608" Oct 30 18:58:49.487748 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.487731 5014 scope.go:122] "RemoveContainer" containerID="babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25" Oct 30 18:58:49.500485 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.500434 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z"] Oct 30 18:58:49.501238 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.500995 5014 scope.go:122] "RemoveContainer" containerID="779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9" Oct 30 18:58:49.506193 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.506148 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-wkl6z"] Oct 30 18:58:49.512977 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.512839 5014 scope.go:122] "RemoveContainer" containerID="be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d" Oct 30 18:58:49.522552 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.522522 5014 scope.go:122] "RemoveContainer" containerID="2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608" Oct 30 18:58:49.522855 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:49.522834 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608\": container with ID starting with 2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608 not found: ID does not exist" containerID="2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608" Oct 30 18:58:49.522933 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.522866 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608"} err="failed to get container status \"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608\": rpc error: code = NotFound desc = could not find container \"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608\": container with ID starting with 2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608 not found: ID does not exist" Oct 30 18:58:49.522933 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.522898 5014 scope.go:122] "RemoveContainer" containerID="babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25" Oct 30 18:58:49.523161 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:49.523143 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25\": container with ID starting with babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25 not found: ID does not exist" containerID="babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25" Oct 30 18:58:49.523204 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.523170 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25"} err="failed to get container status \"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25\": rpc error: code = NotFound desc = could not find container \"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25\": container with ID starting with babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25 not found: ID does not exist" Oct 30 18:58:49.523204 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.523185 5014 scope.go:122] "RemoveContainer" containerID="779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9" Oct 30 18:58:49.523424 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:49.523406 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9\": container with ID starting with 779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9 not found: ID does not exist" containerID="779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9" Oct 30 18:58:49.523477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.523431 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9"} err="failed to get container status \"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9\": rpc error: code = NotFound desc = could not find container \"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9\": container with ID starting with 779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9 not found: ID does not exist" Oct 30 18:58:49.523477 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.523446 5014 scope.go:122] "RemoveContainer" containerID="be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d" Oct 30 18:58:49.523702 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:49.523684 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d\": container with ID starting with be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d not found: ID does not exist" containerID="be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d" Oct 30 18:58:49.523786 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.523708 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d"} err="failed to get container status \"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d\": rpc error: code = NotFound desc = could not find container \"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d\": container with ID starting with be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d not found: ID does not exist" Oct 30 18:58:49.523786 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.523727 5014 scope.go:122] "RemoveContainer" containerID="2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608" Oct 30 18:58:49.524034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.524017 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608"} err="failed to get container status \"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608\": rpc error: code = NotFound desc = could not find container \"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608\": container with ID starting with 2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608 not found: ID does not exist" Oct 30 18:58:49.524101 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.524034 5014 scope.go:122] "RemoveContainer" containerID="babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25" Oct 30 18:58:49.524294 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.524277 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25"} err="failed to get container status \"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25\": rpc error: code = NotFound desc = could not find container \"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25\": container with ID starting with babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25 not found: ID does not exist" Oct 30 18:58:49.524343 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.524304 5014 scope.go:122] "RemoveContainer" containerID="779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9" Oct 30 18:58:49.524546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.524530 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9"} err="failed to get container status \"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9\": rpc error: code = NotFound desc = could not find container \"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9\": container with ID starting with 779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9 not found: ID does not exist" Oct 30 18:58:49.524546 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.524545 5014 scope.go:122] "RemoveContainer" containerID="be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d" Oct 30 18:58:49.524787 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.524768 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d"} err="failed to get container status \"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d\": rpc error: code = NotFound desc = could not find container \"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d\": container with ID starting with be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d not found: ID does not exist" Oct 30 18:58:49.524862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.524789 5014 scope.go:122] "RemoveContainer" containerID="2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608" Oct 30 18:58:49.525031 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.525013 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608"} err="failed to get container status \"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608\": rpc error: code = NotFound desc = could not find container \"2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608\": container with ID starting with 2665bdc790f8f24ebe46a3e6dddcf07bf72b2066baed0935fd61d1a44ff22608 not found: ID does not exist" Oct 30 18:58:49.525081 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.525032 5014 scope.go:122] "RemoveContainer" containerID="babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25" Oct 30 18:58:49.525250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.525231 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25"} err="failed to get container status \"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25\": rpc error: code = NotFound desc = could not find container \"babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25\": container with ID starting with babf0f1d9dfab496cfdea3eb360893c4c1fab8ad5ee52fa6d2c2e896af71bf25 not found: ID does not exist" Oct 30 18:58:49.525320 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.525252 5014 scope.go:122] "RemoveContainer" containerID="779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9" Oct 30 18:58:49.525489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.525474 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9"} err="failed to get container status \"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9\": rpc error: code = NotFound desc = could not find container \"779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9\": container with ID starting with 779b93f3272e18db51887a26a6e3d50a6b1545fe77c20942f9f9743fcebf9ec9 not found: ID does not exist" Oct 30 18:58:49.525548 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.525489 5014 scope.go:122] "RemoveContainer" containerID="be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d" Oct 30 18:58:49.525749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.525731 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d"} err="failed to get container status \"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d\": rpc error: code = NotFound desc = could not find container \"be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d\": container with ID starting with be3b048da1e5e4c6d2aa50a9c1bfb0311fb2781ec450eba0348042ff1507940d not found: ID does not exist" Oct 30 18:58:49.525814 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.525749 5014 scope.go:122] "RemoveContainer" containerID="13ea1441eb9ede52887be0a04c820ca0d874868977098de9d3b206ab80f4c5bd" Oct 30 18:58:49.538460 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.538436 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv"] Oct 30 18:58:49.541953 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.541932 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-ttnsv"] Oct 30 18:58:49.577738 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.577583 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:58:49.583875 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.583854 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:58:49.611476 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.611458 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:49.616681 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.616663 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs"] Oct 30 18:58:49.620481 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.620464 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-zclqs"] Oct 30 18:58:49.710857 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.710822 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4882ef1c-74d9-4a5e-9481-32461fd011f7-kube-api-access-dnt76\" (UniqueName: \"kubernetes.io/projected/4882ef1c-74d9-4a5e-9481-32461fd011f7-kube-api-access-dnt76\") pod \"4882ef1c-74d9-4a5e-9481-32461fd011f7\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " Oct 30 18:58:49.711040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.710878 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-data-science-collector-ta-client-cert\") pod \"4882ef1c-74d9-4a5e-9481-32461fd011f7\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " Oct 30 18:58:49.711040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.710969 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-monitoring-tls-certs\") pod \"4882ef1c-74d9-4a5e-9481-32461fd011f7\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " Oct 30 18:58:49.711040 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.711002 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-tls-certs\") pod \"4882ef1c-74d9-4a5e-9481-32461fd011f7\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " Oct 30 18:58:49.711204 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.711060 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4882ef1c-74d9-4a5e-9481-32461fd011f7-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4882ef1c-74d9-4a5e-9481-32461fd011f7-otc-internal\") pod \"4882ef1c-74d9-4a5e-9481-32461fd011f7\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " Oct 30 18:58:49.711204 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.711106 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-otlp-tls-certs\") pod \"4882ef1c-74d9-4a5e-9481-32461fd011f7\" (UID: \"4882ef1c-74d9-4a5e-9481-32461fd011f7\") " Oct 30 18:58:49.713737 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.711968 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4882ef1c-74d9-4a5e-9481-32461fd011f7-otc-internal" pod "4882ef1c-74d9-4a5e-9481-32461fd011f7" (UID: "4882ef1c-74d9-4a5e-9481-32461fd011f7"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:49.723143 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.723110 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-data-science-collector-ta-client-cert" pod "4882ef1c-74d9-4a5e-9481-32461fd011f7" (UID: "4882ef1c-74d9-4a5e-9481-32461fd011f7"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.723503 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.723461 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-tls-certs" pod "4882ef1c-74d9-4a5e-9481-32461fd011f7" (UID: "4882ef1c-74d9-4a5e-9481-32461fd011f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.725011 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.724987 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4882ef1c-74d9-4a5e-9481-32461fd011f7-kube-api-access-dnt76" pod "4882ef1c-74d9-4a5e-9481-32461fd011f7" (UID: "4882ef1c-74d9-4a5e-9481-32461fd011f7"). InnerVolumeSpecName "kube-api-access-dnt76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:49.725122 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.724984 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-otlp-tls-certs" pod "4882ef1c-74d9-4a5e-9481-32461fd011f7" (UID: "4882ef1c-74d9-4a5e-9481-32461fd011f7"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.725613 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.725575 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-monitoring-tls-certs" pod "4882ef1c-74d9-4a5e-9481-32461fd011f7" (UID: "4882ef1c-74d9-4a5e-9481-32461fd011f7"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.820116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.820091 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4882ef1c-74d9-4a5e-9481-32461fd011f7-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.820116 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.820117 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.820277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.820130 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnt76\" (UniqueName: \"kubernetes.io/projected/4882ef1c-74d9-4a5e-9481-32461fd011f7-kube-api-access-dnt76\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.820277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.820145 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.820277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.820158 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.820277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:49.820170 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4882ef1c-74d9-4a5e-9481-32461fd011f7-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:50.412147 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:50.412116 5014 generic.go:356] "Generic (PLEG): container finished" podID="4882ef1c-74d9-4a5e-9481-32461fd011f7" containerID="bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc" exitCode=0 Oct 30 18:58:50.412697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:50.412157 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4882ef1c-74d9-4a5e-9481-32461fd011f7","Type":"ContainerDied","Data":"bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc"} Oct 30 18:58:50.412697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:50.412174 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:58:50.412697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:50.412206 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4882ef1c-74d9-4a5e-9481-32461fd011f7","Type":"ContainerDied","Data":"993df3585b649afc63d9418a8bd34c6e2c304ce62c4836744f973351d4c05a76"} Oct 30 18:58:50.412697 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:50.412236 5014 scope.go:122] "RemoveContainer" containerID="bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc" Oct 30 18:58:50.421644 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:50.421628 5014 scope.go:122] "RemoveContainer" containerID="bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc" Oct 30 18:58:50.421922 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:50.421903 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc\": container with ID starting with bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc not found: ID does not exist" containerID="bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc" Oct 30 18:58:50.421969 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:50.421929 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc"} err="failed to get container status \"bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc\": rpc error: code = NotFound desc = could not find container \"bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc\": container with ID starting with bdeefd05ce8cbcbce8dcad23eb95fe7f53e53fd2c6f6438846b654825e4f93fc not found: ID does not exist" Oct 30 18:58:50.462246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:50.462226 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:50.474786 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:50.474732 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:58:51.140847 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:51.140820 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" path="/var/lib/kubelet/pods/09ee8f75-b086-4e90-acbc-01c763cce041/volumes" Oct 30 18:58:51.141268 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:51.141255 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4882ef1c-74d9-4a5e-9481-32461fd011f7" path="/var/lib/kubelet/pods/4882ef1c-74d9-4a5e-9481-32461fd011f7/volumes" Oct 30 18:58:51.141626 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:51.141613 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841093e5-a2af-4af3-8635-d9bb3e914128" path="/var/lib/kubelet/pods/841093e5-a2af-4af3-8635-d9bb3e914128/volumes" Oct 30 18:58:51.142300 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:51.142285 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" path="/var/lib/kubelet/pods/b3e48898-eb53-4585-9396-0fcd7ac596b2/volumes" Oct 30 18:58:51.142714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:51.142701 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba31f50f-d878-472b-9986-03903b6472da" path="/var/lib/kubelet/pods/ba31f50f-d878-472b-9986-03903b6472da/volumes" Oct 30 18:58:51.162560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:51.162533 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.161:9093/-/ready\": dial tcp 10.132.0.161:9093: connect: connection refused" Oct 30 18:58:56.162884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:56.162833 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.161:9093/-/ready\": dial tcp 10.132.0.161:9093: connect: connection refused" Oct 30 18:58:58.942178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:58.942154 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:58:59.086043 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.086012 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-volume\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-volume\") pod \"940e125c-a268-4c06-87d0-6bbb63b03b1d\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " Oct 30 18:58:59.086174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.086060 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-kube-api-access-qppcq\" (UniqueName: \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-kube-api-access-qppcq\") pod \"940e125c-a268-4c06-87d0-6bbb63b03b1d\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " Oct 30 18:58:59.086174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.086094 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-alertmanager-data-science-monitoringstack-db\") pod \"940e125c-a268-4c06-87d0-6bbb63b03b1d\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " Oct 30 18:58:59.086174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.086122 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-out\") pod \"940e125c-a268-4c06-87d0-6bbb63b03b1d\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " Oct 30 18:58:59.086174 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.086148 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-web-config\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-web-config\") pod \"940e125c-a268-4c06-87d0-6bbb63b03b1d\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " Oct 30 18:58:59.086409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.086193 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-tls-assets\" (UniqueName: \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-tls-assets\") pod \"940e125c-a268-4c06-87d0-6bbb63b03b1d\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " Oct 30 18:58:59.086409 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.086225 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-cluster-tls-config\") pod \"940e125c-a268-4c06-87d0-6bbb63b03b1d\" (UID: \"940e125c-a268-4c06-87d0-6bbb63b03b1d\") " Oct 30 18:58:59.086522 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.086449 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-alertmanager-data-science-monitoringstack-db" pod "940e125c-a268-4c06-87d0-6bbb63b03b1d" (UID: "940e125c-a268-4c06-87d0-6bbb63b03b1d"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:59.088402 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.088365 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-out" pod "940e125c-a268-4c06-87d0-6bbb63b03b1d" (UID: "940e125c-a268-4c06-87d0-6bbb63b03b1d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:59.088796 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.088770 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-volume" pod "940e125c-a268-4c06-87d0-6bbb63b03b1d" (UID: "940e125c-a268-4c06-87d0-6bbb63b03b1d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:59.088796 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.088781 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-kube-api-access-qppcq" pod "940e125c-a268-4c06-87d0-6bbb63b03b1d" (UID: "940e125c-a268-4c06-87d0-6bbb63b03b1d"). InnerVolumeSpecName "kube-api-access-qppcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:59.089084 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.089061 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-tls-assets" pod "940e125c-a268-4c06-87d0-6bbb63b03b1d" (UID: "940e125c-a268-4c06-87d0-6bbb63b03b1d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:59.092724 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.092700 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-cluster-tls-config" pod "940e125c-a268-4c06-87d0-6bbb63b03b1d" (UID: "940e125c-a268-4c06-87d0-6bbb63b03b1d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:59.100279 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.100257 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-web-config" pod "940e125c-a268-4c06-87d0-6bbb63b03b1d" (UID: "940e125c-a268-4c06-87d0-6bbb63b03b1d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:59.187078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.187056 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:59.187078 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.187076 5014 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-cluster-tls-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:59.187203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.187085 5014 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-volume\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:59.187203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.187094 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qppcq\" (UniqueName: \"kubernetes.io/projected/940e125c-a268-4c06-87d0-6bbb63b03b1d-kube-api-access-qppcq\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:59.187203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.187105 5014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:59.187203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.187113 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/940e125c-a268-4c06-87d0-6bbb63b03b1d-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:59.187203 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.187140 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/940e125c-a268-4c06-87d0-6bbb63b03b1d-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:58:59.461045 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.460959 5014 generic.go:356] "Generic (PLEG): container finished" podID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerID="64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f" exitCode=0 Oct 30 18:58:59.461277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.461110 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"940e125c-a268-4c06-87d0-6bbb63b03b1d","Type":"ContainerDied","Data":"64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f"} Oct 30 18:58:59.461277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.461166 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"940e125c-a268-4c06-87d0-6bbb63b03b1d","Type":"ContainerDied","Data":"3fb5f9d91c3802af7d6840586b582ff41a3a9ec55fc353a980342cd620c56b46"} Oct 30 18:58:59.461277 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.461202 5014 scope.go:122] "RemoveContainer" containerID="ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5" Oct 30 18:58:59.461453 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.461343 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:58:59.472730 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.472681 5014 scope.go:122] "RemoveContainer" containerID="64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f" Oct 30 18:58:59.482474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.482390 5014 scope.go:122] "RemoveContainer" containerID="1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b" Oct 30 18:58:59.493495 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.493417 5014 scope.go:122] "RemoveContainer" containerID="ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5" Oct 30 18:58:59.496964 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:59.496933 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5\": container with ID starting with ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5 not found: ID does not exist" containerID="ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5" Oct 30 18:58:59.497080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.496974 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5"} err="failed to get container status \"ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5\": rpc error: code = NotFound desc = could not find container \"ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5\": container with ID starting with ece7aca2522854778767b7ceb0d7452f5fb8a85f79875096c198f31f5116c1a5 not found: ID does not exist" Oct 30 18:58:59.497080 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.496995 5014 scope.go:122] "RemoveContainer" containerID="64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f" Oct 30 18:58:59.497468 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:59.497447 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f\": container with ID starting with 64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f not found: ID does not exist" containerID="64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f" Oct 30 18:58:59.497555 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.497472 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f"} err="failed to get container status \"64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f\": rpc error: code = NotFound desc = could not find container \"64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f\": container with ID starting with 64612225a1a51ffe131be9323229a9e585be60405cdf4c0ecf7b387442813d3f not found: ID does not exist" Oct 30 18:58:59.497555 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.497489 5014 scope.go:122] "RemoveContainer" containerID="1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b" Oct 30 18:58:59.497865 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:58:59.497778 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b\": container with ID starting with 1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b not found: ID does not exist" containerID="1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b" Oct 30 18:58:59.497865 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.497798 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b"} err="failed to get container status \"1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b\": rpc error: code = NotFound desc = could not find container \"1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b\": container with ID starting with 1b9f762211a89afbacb82501a69d26325b4c97545b68c7b8448ef882694aad0b not found: ID does not exist" Oct 30 18:58:59.521391 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.516012 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:58:59.521391 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.521385 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:58:59.619946 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.619919 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx"] Oct 30 18:58:59.620228 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620216 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="config-reloader" Oct 30 18:58:59.620273 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620229 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="config-reloader" Oct 30 18:58:59.620273 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620243 5014 cpu_manager.go:414] "containerMap: removing container" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="init-config-reloader" Oct 30 18:58:59.620273 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620248 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="init-config-reloader" Oct 30 18:58:59.620273 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620257 5014 cpu_manager.go:414] "containerMap: removing container" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="kube-rbac-proxy" Oct 30 18:58:59.620273 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620262 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="kube-rbac-proxy" Oct 30 18:58:59.620273 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620269 5014 cpu_manager.go:414] "containerMap: removing container" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="alertmanager" Oct 30 18:58:59.620273 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620274 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="alertmanager" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620281 5014 cpu_manager.go:414] "containerMap: removing container" podUID="841093e5-a2af-4af3-8635-d9bb3e914128" containerName="kube-rbac-proxy" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620286 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841093e5-a2af-4af3-8635-d9bb3e914128" containerName="kube-rbac-proxy" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620295 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="thanos-sidecar" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620300 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="thanos-sidecar" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620311 5014 cpu_manager.go:414] "containerMap: removing container" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="prom-label-proxy" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620317 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="prom-label-proxy" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620335 5014 cpu_manager.go:414] "containerMap: removing container" podUID="ba31f50f-d878-472b-9986-03903b6472da" containerName="thanos-querier" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620340 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba31f50f-d878-472b-9986-03903b6472da" containerName="thanos-querier" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620346 5014 cpu_manager.go:414] "containerMap: removing container" podUID="4882ef1c-74d9-4a5e-9481-32461fd011f7" containerName="otc-container" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620351 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4882ef1c-74d9-4a5e-9481-32461fd011f7" containerName="otc-container" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620361 5014 cpu_manager.go:414] "containerMap: removing container" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="method-gate" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620365 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="method-gate" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620371 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="init-config-reloader" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620377 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="init-config-reloader" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620384 5014 cpu_manager.go:414] "containerMap: removing container" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="config-reloader" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620388 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="config-reloader" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620395 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="prometheus" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620399 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="prometheus" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620447 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="method-gate" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620454 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="config-reloader" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620461 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="prometheus" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620468 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="config-reloader" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620474 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="kube-rbac-proxy" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620478 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" containerName="alertmanager" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620484 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="841093e5-a2af-4af3-8635-d9bb3e914128" containerName="kube-rbac-proxy" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620490 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e48898-eb53-4585-9396-0fcd7ac596b2" containerName="thanos-sidecar" Oct 30 18:58:59.620487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620497 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ee8f75-b086-4e90-acbc-01c763cce041" containerName="prom-label-proxy" Oct 30 18:58:59.621447 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620502 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba31f50f-d878-472b-9986-03903b6472da" containerName="thanos-querier" Oct 30 18:58:59.621447 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.620508 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="4882ef1c-74d9-4a5e-9481-32461fd011f7" containerName="otc-container" Oct 30 18:58:59.625076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.625060 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.643884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.643859 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx"] Oct 30 18:58:59.645717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.645680 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:58:59.645717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.645680 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 18:58:59.645717 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.645686 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:58:59.645928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.645697 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-cvxgk\"" Oct 30 18:58:59.645928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.645728 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:58:59.645928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.645748 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 18:58:59.645928 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.645818 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:58:59.662059 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.662037 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74"] Oct 30 18:58:59.665805 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.665788 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.686575 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.686555 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74"] Oct 30 18:58:59.745749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.745693 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 18:58:59.745749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.745734 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-rnljh\"" Oct 30 18:58:59.745943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.745930 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 18:58:59.745985 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.745936 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 18:58:59.792068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.792048 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.792180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.792079 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.792180 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.792101 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwlct\" (UniqueName: \"kubernetes.io/projected/9207d774-f57c-4091-81a2-f195690744d1-kube-api-access-rwlct\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.792259 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.792210 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9207d774-f57c-4091-81a2-f195690744d1-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.792259 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.792236 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82lbv\" (UniqueName: \"kubernetes.io/projected/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-api-access-82lbv\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.792337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.792259 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.792337 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.792291 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.792411 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.792398 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.792451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.792422 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.792451 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.792441 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.892856 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.892835 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.892951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.892865 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwlct\" (UniqueName: \"kubernetes.io/projected/9207d774-f57c-4091-81a2-f195690744d1-kube-api-access-rwlct\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.892951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.892895 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9207d774-f57c-4091-81a2-f195690744d1-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.892951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.892912 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82lbv\" (UniqueName: \"kubernetes.io/projected/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-api-access-82lbv\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.892951 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.892932 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.893164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.893057 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.893164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.893101 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.893164 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.893135 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.893385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.893167 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.893385 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.893271 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.893584 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.893483 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.893687 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.893661 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.893807 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.893786 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.894223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.894181 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.895749 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.895726 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9207d774-f57c-4091-81a2-f195690744d1-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.895828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.895777 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.895867 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.895833 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.896206 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.896184 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.915759 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.915736 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82lbv\" (UniqueName: \"kubernetes.io/projected/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-api-access-82lbv\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7lswx\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.915927 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.915910 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwlct\" (UniqueName: \"kubernetes.io/projected/9207d774-f57c-4091-81a2-f195690744d1-kube-api-access-rwlct\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-nbk74\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:58:59.937703 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.937681 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:58:59.983332 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:58:59.983306 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:59:00.233291 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:00.233263 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx"] Oct 30 18:59:00.237977 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:59:00.237922 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda585f6c7_b0bc_42bd_9cfa_53b7a9d26138.slice/crio-e23b94f2e991ea2b2462c0eab18883b6c7d599fece90f7611ead6b499e4bea83 WatchSource:0}: Error finding container e23b94f2e991ea2b2462c0eab18883b6c7d599fece90f7611ead6b499e4bea83: Status 404 returned error can't find the container with id e23b94f2e991ea2b2462c0eab18883b6c7d599fece90f7611ead6b499e4bea83 Oct 30 18:59:00.288641 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:00.288614 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74"] Oct 30 18:59:00.292954 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:59:00.292931 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9207d774_f57c_4091_81a2_f195690744d1.slice/crio-c773c454aef3e07cc35ffb6c3867ced66d898e06a353d8e0368c8ed9875b100e WatchSource:0}: Error finding container c773c454aef3e07cc35ffb6c3867ced66d898e06a353d8e0368c8ed9875b100e: Status 404 returned error can't find the container with id c773c454aef3e07cc35ffb6c3867ced66d898e06a353d8e0368c8ed9875b100e Oct 30 18:59:00.468335 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:00.468301 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" event={"ID":"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138","Type":"ContainerStarted","Data":"e23b94f2e991ea2b2462c0eab18883b6c7d599fece90f7611ead6b499e4bea83"} Oct 30 18:59:00.470072 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:00.470049 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" event={"ID":"9207d774-f57c-4091-81a2-f195690744d1","Type":"ContainerStarted","Data":"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13"} Oct 30 18:59:00.470178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:00.470103 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" event={"ID":"9207d774-f57c-4091-81a2-f195690744d1","Type":"ContainerStarted","Data":"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918"} Oct 30 18:59:00.470178 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:00.470127 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" event={"ID":"9207d774-f57c-4091-81a2-f195690744d1","Type":"ContainerStarted","Data":"c773c454aef3e07cc35ffb6c3867ced66d898e06a353d8e0368c8ed9875b100e"} Oct 30 18:59:01.144179 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.144147 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940e125c-a268-4c06-87d0-6bbb63b03b1d" path="/var/lib/kubelet/pods/940e125c-a268-4c06-87d0-6bbb63b03b1d/volumes" Oct 30 18:59:01.144936 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.144643 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:01.148945 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.148921 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.166382 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.166355 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:01.168562 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.167956 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:59:01.168562 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.168049 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:59:01.170745 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.170726 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-bhcr4\"" Oct 30 18:59:01.268020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.267972 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:59:01.268020 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.268004 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:59:01.268228 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.268105 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:59:01.305587 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.305539 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.305776 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.305671 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.305834 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.305774 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9bb\" (UniqueName: \"kubernetes.io/projected/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-kube-api-access-gt9bb\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.305884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.305828 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.306083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.306065 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.306126 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.306112 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.406813 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.406743 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.406813 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.406779 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.406813 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.406799 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.407068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.406815 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.407068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.406938 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9bb\" (UniqueName: \"kubernetes.io/projected/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-kube-api-access-gt9bb\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.407068 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.406991 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.407581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.407507 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.409624 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.409573 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.409624 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.409617 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.409770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.409655 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.409770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.409700 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.433679 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.433654 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9bb\" (UniqueName: \"kubernetes.io/projected/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-kube-api-access-gt9bb\") pod \"data-science-collector-collector-0\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.463940 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.463917 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:01.480982 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.480953 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" event={"ID":"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138","Type":"ContainerStarted","Data":"af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499"} Oct 30 18:59:01.481240 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.481223 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:59:01.485353 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.485331 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" event={"ID":"9207d774-f57c-4091-81a2-f195690744d1","Type":"ContainerStarted","Data":"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165"} Oct 30 18:59:01.573718 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.573666 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" podStartSLOduration=2.573652077 podStartE2EDuration="2.573652077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:01.523336378 +0000 UTC m=+3754.511676867" watchObservedRunningTime="2026-10-30 18:59:01.573652077 +0000 UTC m=+3754.561992563" Oct 30 18:59:01.573883 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.573829 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" podStartSLOduration=2.573824701 podStartE2EDuration="2.573824701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:01.57236352 +0000 UTC m=+3754.560704007" watchObservedRunningTime="2026-10-30 18:59:01.573824701 +0000 UTC m=+3754.562165188" Oct 30 18:59:01.772632 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.772535 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:01.777060 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:59:01.777027 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1a9fa9_a4dc_4b80_a53e_9443e34358df.slice/crio-a3d819055d5768bf209c966a8e255f34447d2b3789d8b1837b50f65ea5483e29 WatchSource:0}: Error finding container a3d819055d5768bf209c966a8e255f34447d2b3789d8b1837b50f65ea5483e29: Status 404 returned error can't find the container with id a3d819055d5768bf209c966a8e255f34447d2b3789d8b1837b50f65ea5483e29 Oct 30 18:59:01.779535 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.779499 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:59:01.779621 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:01.779581 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:59:02.492518 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:02.492477 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fc1a9fa9-a4dc-4b80-a53e-9443e34358df","Type":"ContainerStarted","Data":"572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541"} Oct 30 18:59:02.492942 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:02.492541 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fc1a9fa9-a4dc-4b80-a53e-9443e34358df","Type":"ContainerStarted","Data":"a3d819055d5768bf209c966a8e255f34447d2b3789d8b1837b50f65ea5483e29"} Oct 30 18:59:02.537352 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:02.537304 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.537291207 podStartE2EDuration="1.537291207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:02.536357446 +0000 UTC m=+3755.524697934" watchObservedRunningTime="2026-10-30 18:59:02.537291207 +0000 UTC m=+3755.525631696" Oct 30 18:59:07.495328 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:07.495299 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:59:07.942472 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:07.942440 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g"] Oct 30 18:59:07.946181 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:07.946153 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" Oct 30 18:59:07.960486 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:07.960465 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-5mtd6\"" Oct 30 18:59:07.965425 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:07.965397 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g"] Oct 30 18:59:08.057006 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.056980 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hscn8\" (UniqueName: \"kubernetes.io/projected/ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e-kube-api-access-hscn8\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g\" (UID: \"ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" Oct 30 18:59:08.157840 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.157814 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hscn8\" (UniqueName: \"kubernetes.io/projected/ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e-kube-api-access-hscn8\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g\" (UID: \"ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" Oct 30 18:59:08.181670 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.181642 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hscn8\" (UniqueName: \"kubernetes.io/projected/ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e-kube-api-access-hscn8\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g\" (UID: \"ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" Oct 30 18:59:08.271743 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.271691 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" Oct 30 18:59:08.568607 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.568573 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g"] Oct 30 18:59:08.573319 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:59:08.573282 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae5c7eee_8b8f_4e8a_81c1_3f291319ac8e.slice/crio-fcc3dcb6600ce5391b96945e671a95375698c586d3367b78e3899e439f90df9d WatchSource:0}: Error finding container fcc3dcb6600ce5391b96945e671a95375698c586d3367b78e3899e439f90df9d: Status 404 returned error can't find the container with id fcc3dcb6600ce5391b96945e671a95375698c586d3367b78e3899e439f90df9d Oct 30 18:59:08.782404 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.782366 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:59:08.786273 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.786249 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:08.801967 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.801948 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:59:08.805753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.805727 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.811775 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.811738 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:59:08.812537 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.812435 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:59:08.816580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.812719 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:59:08.816580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.812799 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:59:08.816580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.814994 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:59:08.816580 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.815072 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-sb2qr\"" Oct 30 18:59:08.852145 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.852081 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:59:08.908560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.908510 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:59:08.908560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.908519 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:59:08.908560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.908540 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:59:08.908560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.908524 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:59:08.908560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.908556 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:59:08.908881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.908537 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:59:08.908881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.908630 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:59:08.908881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.908729 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-wdmbl\"" Oct 30 18:59:08.908881 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.908756 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:59:08.964788 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.964765 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.964902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.964790 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:08.964902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.964811 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.964902 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.964837 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8z4r\" (UniqueName: \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-kube-api-access-q8z4r\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.965039 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.964995 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.965039 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965023 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.965125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965051 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.965171 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965129 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.965171 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965153 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:08.965250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965176 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:08.965250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965196 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.965331 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965273 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.965331 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965302 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.965331 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965322 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.965452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965342 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:08.965452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965366 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:08.965452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965393 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:08.965452 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965425 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:08.965630 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:08.965467 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6zh\" (UniqueName: \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-kube-api-access-vg6zh\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.066318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066293 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.066424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066323 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.066424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066343 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.066424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066362 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.066424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066385 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.066700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066504 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.066700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066551 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.066700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066617 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.066700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066647 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.066700 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066677 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.066983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066725 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.066983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066764 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.066983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066792 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.066983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066820 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.066983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066867 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6zh\" (UniqueName: \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-kube-api-access-vg6zh\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.066983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066914 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.066983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066941 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.066983 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.066972 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.067424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.067020 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8z4r\" (UniqueName: \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-kube-api-access-q8z4r\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.067424 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.067060 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.069734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.069696 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.069862 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.069838 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.069950 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.069915 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.070023 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.070004 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.070092 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.070018 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.070092 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.070020 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.070325 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.070298 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.070647 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.070624 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.071428 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.071396 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.071581 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.071559 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.071719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.071695 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.072434 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.072408 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.072534 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.072488 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.072664 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.072642 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.072755 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.072732 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.073064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.073046 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.089872 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.089851 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8z4r\" (UniqueName: \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-kube-api-access-q8z4r\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:09.089956 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.089923 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6zh\" (UniqueName: \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-kube-api-access-vg6zh\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.103863 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.103817 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:09.415714 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.415625 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:59:09.420583 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:59:09.420556 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0282b81d_e88f_4833_8abf_4bac747bce1e.slice/crio-e5bdcb41e61ed843a09b62edae3d1569be9485655aa986698f8cc9a3d1e52fa8 WatchSource:0}: Error finding container e5bdcb41e61ed843a09b62edae3d1569be9485655aa986698f8cc9a3d1e52fa8: Status 404 returned error can't find the container with id e5bdcb41e61ed843a09b62edae3d1569be9485655aa986698f8cc9a3d1e52fa8 Oct 30 18:59:09.531083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.531055 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"0282b81d-e88f-4833-8abf-4bac747bce1e","Type":"ContainerStarted","Data":"e5bdcb41e61ed843a09b62edae3d1569be9485655aa986698f8cc9a3d1e52fa8"} Oct 30 18:59:09.532531 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.532509 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" event={"ID":"ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e","Type":"ContainerStarted","Data":"775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662"} Oct 30 18:59:09.532637 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.532540 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" event={"ID":"ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e","Type":"ContainerStarted","Data":"fcc3dcb6600ce5391b96945e671a95375698c586d3367b78e3899e439f90df9d"} Oct 30 18:59:09.571666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:09.571629 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" podStartSLOduration=2.571616031 podStartE2EDuration="2.571616031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:09.570152038 +0000 UTC m=+3762.558492526" watchObservedRunningTime="2026-10-30 18:59:09.571616031 +0000 UTC m=+3762.559956514" Oct 30 18:59:10.542288 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:10.542215 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"0282b81d-e88f-4833-8abf-4bac747bce1e","Type":"ContainerStarted","Data":"c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3"} Oct 30 18:59:13.331371 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:13.331338 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:59:13.558345 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:13.558313 5014 generic.go:356] "Generic (PLEG): container finished" podID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerID="c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3" exitCode=0 Oct 30 18:59:13.558491 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:13.558386 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"0282b81d-e88f-4833-8abf-4bac747bce1e","Type":"ContainerDied","Data":"c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3"} Oct 30 18:59:14.566568 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:14.566528 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"0282b81d-e88f-4833-8abf-4bac747bce1e","Type":"ContainerStarted","Data":"16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7"} Oct 30 18:59:14.566932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:14.566611 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"0282b81d-e88f-4833-8abf-4bac747bce1e","Type":"ContainerStarted","Data":"06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35"} Oct 30 18:59:14.566932 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:14.566792 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:14.613841 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:14.613793 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.128821464 podStartE2EDuration="6.613778815s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484957351s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:14.612844756 +0000 UTC m=+3767.601185243" watchObservedRunningTime="2026-10-30 18:59:14.613778815 +0000 UTC m=+3767.602119356" Oct 30 18:59:17.126319 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:17.126279 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:17.155015 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:17.154988 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:17.226671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:17.226644 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:17.226830 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:17.226768 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:17.241291 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:17.241269 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:17.344282 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:17.344251 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/26c702426f1aeab2e9e6116e994cbd362b1230f690bb6d32002e36ee94fa3aec/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:17.371057 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:17.371036 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:17.529158 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:17.529101 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:17.866853 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:59:17.866811 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b3a82c_4d56_4db5_9918_068bbe69d9d3.slice/crio-4c47b393e42502a55b3b8da83e5ca3aebda50434580bdff6f3870cc119aebde8 WatchSource:0}: Error finding container 4c47b393e42502a55b3b8da83e5ca3aebda50434580bdff6f3870cc119aebde8: Status 404 returned error can't find the container with id 4c47b393e42502a55b3b8da83e5ca3aebda50434580bdff6f3870cc119aebde8 Oct 30 18:59:17.875298 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:17.875279 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:59:18.591500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:18.591465 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b7b3a82c-4d56-4db5-9918-068bbe69d9d3","Type":"ContainerStarted","Data":"5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4"} Oct 30 18:59:18.591893 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:18.591522 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b7b3a82c-4d56-4db5-9918-068bbe69d9d3","Type":"ContainerStarted","Data":"4c47b393e42502a55b3b8da83e5ca3aebda50434580bdff6f3870cc119aebde8"} Oct 30 18:59:20.573085 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:20.573053 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:21.607550 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:21.607522 5014 generic.go:356] "Generic (PLEG): container finished" podID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerID="5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4" exitCode=0 Oct 30 18:59:21.607903 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:21.607602 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b7b3a82c-4d56-4db5-9918-068bbe69d9d3","Type":"ContainerDied","Data":"5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4"} Oct 30 18:59:22.615653 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:22.615621 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b7b3a82c-4d56-4db5-9918-068bbe69d9d3","Type":"ContainerStarted","Data":"025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152"} Oct 30 18:59:22.616047 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:22.615684 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b7b3a82c-4d56-4db5-9918-068bbe69d9d3","Type":"ContainerStarted","Data":"2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c"} Oct 30 18:59:22.616047 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:22.615707 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b7b3a82c-4d56-4db5-9918-068bbe69d9d3","Type":"ContainerStarted","Data":"ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8"} Oct 30 18:59:22.680735 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:22.680694 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=11.495081246 podStartE2EDuration="14.680682045s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.185600799s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:22.66885189 +0000 UTC m=+3775.657192378" watchObservedRunningTime="2026-10-30 18:59:22.680682045 +0000 UTC m=+3775.669022531" Oct 30 18:59:27.529513 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:27.529486 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:32.530195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:32.530162 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:32.535887 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:32.535864 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:32.667531 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:32.667505 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:35.702872 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:35.702839 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:35.703300 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:35.703113 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="fc1a9fa9-a4dc-4b80-a53e-9443e34358df" containerName="otc-container" containerID="cri-o://572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541" gracePeriod=30 Oct 30 18:59:38.048256 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.048227 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.190872 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.190805 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-kube-api-access-gt9bb\" (UniqueName: \"kubernetes.io/projected/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-kube-api-access-gt9bb\") pod \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " Oct 30 18:59:38.190872 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.190864 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-data-science-collector-ta-client-cert\") pod \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " Oct 30 18:59:38.191095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.190883 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otlp-tls-certs\") pod \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " Oct 30 18:59:38.191095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.190924 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-monitoring-tls-certs\") pod \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " Oct 30 18:59:38.191095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.190943 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otc-internal\" (UniqueName: \"kubernetes.io/configmap/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otc-internal\") pod \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " Oct 30 18:59:38.191095 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.190970 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-tls-certs\") pod \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\" (UID: \"fc1a9fa9-a4dc-4b80-a53e-9443e34358df\") " Oct 30 18:59:38.191375 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.191346 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otc-internal" pod "fc1a9fa9-a4dc-4b80-a53e-9443e34358df" (UID: "fc1a9fa9-a4dc-4b80-a53e-9443e34358df"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:38.193239 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.193211 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-data-science-collector-ta-client-cert" pod "fc1a9fa9-a4dc-4b80-a53e-9443e34358df" (UID: "fc1a9fa9-a4dc-4b80-a53e-9443e34358df"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:38.193361 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.193239 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-monitoring-tls-certs" pod "fc1a9fa9-a4dc-4b80-a53e-9443e34358df" (UID: "fc1a9fa9-a4dc-4b80-a53e-9443e34358df"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:38.193361 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.193331 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-tls-certs" pod "fc1a9fa9-a4dc-4b80-a53e-9443e34358df" (UID: "fc1a9fa9-a4dc-4b80-a53e-9443e34358df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:38.193622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.193584 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otlp-tls-certs" pod "fc1a9fa9-a4dc-4b80-a53e-9443e34358df" (UID: "fc1a9fa9-a4dc-4b80-a53e-9443e34358df"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:38.193679 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.193645 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-kube-api-access-gt9bb" pod "fc1a9fa9-a4dc-4b80-a53e-9443e34358df" (UID: "fc1a9fa9-a4dc-4b80-a53e-9443e34358df"). InnerVolumeSpecName "kube-api-access-gt9bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:38.292125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.292104 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:38.292125 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.292124 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:38.292246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.292134 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:38.292246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.292142 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt9bb\" (UniqueName: \"kubernetes.io/projected/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-kube-api-access-gt9bb\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:38.292246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.292151 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:38.292246 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.292159 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a9fa9-a4dc-4b80-a53e-9443e34358df-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:38.694564 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.694534 5014 generic.go:356] "Generic (PLEG): container finished" podID="fc1a9fa9-a4dc-4b80-a53e-9443e34358df" containerID="572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541" exitCode=0 Oct 30 18:59:38.694740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.694588 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fc1a9fa9-a4dc-4b80-a53e-9443e34358df","Type":"ContainerDied","Data":"572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541"} Oct 30 18:59:38.694740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.694618 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.694740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.694644 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fc1a9fa9-a4dc-4b80-a53e-9443e34358df","Type":"ContainerDied","Data":"a3d819055d5768bf209c966a8e255f34447d2b3789d8b1837b50f65ea5483e29"} Oct 30 18:59:38.694740 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.694665 5014 scope.go:122] "RemoveContainer" containerID="572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541" Oct 30 18:59:38.704538 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.704516 5014 scope.go:122] "RemoveContainer" containerID="572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541" Oct 30 18:59:38.704921 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:38.704902 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541\": container with ID starting with 572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541 not found: ID does not exist" containerID="572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541" Oct 30 18:59:38.705010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.704928 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541"} err="failed to get container status \"572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541\": rpc error: code = NotFound desc = could not find container \"572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541\": container with ID starting with 572ce6ab2bce763d0762f0080e433be707a9a4a8eda4cc145664d2566c2ab541 not found: ID does not exist" Oct 30 18:59:38.732092 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:38.732074 5014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1a9fa9_a4dc_4b80_a53e_9443e34358df.slice/crio-a3d819055d5768bf209c966a8e255f34447d2b3789d8b1837b50f65ea5483e29\": RecentStats: unable to find data in memory cache]" Oct 30 18:59:38.746502 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.746481 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:38.758376 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.758356 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:38.788155 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.788130 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:38.788643 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.788617 5014 cpu_manager.go:414] "containerMap: removing container" podUID="fc1a9fa9-a4dc-4b80-a53e-9443e34358df" containerName="otc-container" Oct 30 18:59:38.788734 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.788652 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1a9fa9-a4dc-4b80-a53e-9443e34358df" containerName="otc-container" Oct 30 18:59:38.788780 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.788765 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1a9fa9-a4dc-4b80-a53e-9443e34358df" containerName="otc-container" Oct 30 18:59:38.793173 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.793157 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.807852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.807822 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:59:38.807852 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.807831 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:59:38.807995 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.807864 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:59:38.807995 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.807899 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:59:38.807995 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.807939 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:59:38.808096 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.808019 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-bhcr4\"" Oct 30 18:59:38.820650 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.820626 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:38.896791 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.896763 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.896910 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.896792 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.896965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.896910 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.896965 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.896962 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.897057 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.896988 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q642p\" (UniqueName: \"kubernetes.io/projected/c23bf958-c5a3-4133-ad82-4b7631a5ae73-kube-api-access-q642p\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.897057 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.897010 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.997706 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.997648 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.997706 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.997675 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.997706 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.997700 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q642p\" (UniqueName: \"kubernetes.io/projected/c23bf958-c5a3-4133-ad82-4b7631a5ae73-kube-api-access-q642p\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.997944 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.997729 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.997944 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.997763 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.997944 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.997789 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:38.998704 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:38.998681 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:39.000374 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.000351 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:39.000528 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.000511 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:39.000680 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.000659 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:39.000728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.000665 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:39.020389 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.020371 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q642p\" (UniqueName: \"kubernetes.io/projected/c23bf958-c5a3-4133-ad82-4b7631a5ae73-kube-api-access-q642p\") pod \"data-science-collector-collector-0\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:39.106551 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.106532 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:39.143083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.143053 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1a9fa9-a4dc-4b80-a53e-9443e34358df" path="/var/lib/kubelet/pods/fc1a9fa9-a4dc-4b80-a53e-9443e34358df/volumes" Oct 30 18:59:39.410034 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.410008 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:39.413578 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:59:39.413552 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23bf958_c5a3_4133_ad82_4b7631a5ae73.slice/crio-a18a597a765d18619663e9baf8b14309015b11de85baaed834f10b611513bbd3 WatchSource:0}: Error finding container a18a597a765d18619663e9baf8b14309015b11de85baaed834f10b611513bbd3: Status 404 returned error can't find the container with id a18a597a765d18619663e9baf8b14309015b11de85baaed834f10b611513bbd3 Oct 30 18:59:39.415723 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.415696 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:59:39.415799 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.415755 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:59:39.701051 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.700967 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c23bf958-c5a3-4133-ad82-4b7631a5ae73","Type":"ContainerStarted","Data":"6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f"} Oct 30 18:59:39.701051 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.701023 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c23bf958-c5a3-4133-ad82-4b7631a5ae73","Type":"ContainerStarted","Data":"a18a597a765d18619663e9baf8b14309015b11de85baaed834f10b611513bbd3"} Oct 30 18:59:39.744869 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:39.744819 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.744806756 podStartE2EDuration="1.744806756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:39.742542598 +0000 UTC m=+3792.730883085" watchObservedRunningTime="2026-10-30 18:59:39.744806756 +0000 UTC m=+3792.733147243" Oct 30 18:59:43.334547 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:43.334511 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 18:59:51.764311 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:51.764266 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:51.764990 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:51.764494 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="c23bf958-c5a3-4133-ad82-4b7631a5ae73" containerName="otc-container" containerID="cri-o://6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f" gracePeriod=30 Oct 30 18:59:54.115585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.115561 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:54.218084 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.218023 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otlp-tls-certs\") pod \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " Oct 30 18:59:54.218084 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.218078 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23bf958-c5a3-4133-ad82-4b7631a5ae73-kube-api-access-q642p\" (UniqueName: \"kubernetes.io/projected/c23bf958-c5a3-4133-ad82-4b7631a5ae73-kube-api-access-q642p\") pod \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " Oct 30 18:59:54.218250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.218109 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-tls-certs\") pod \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " Oct 30 18:59:54.218250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.218147 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otc-internal\") pod \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " Oct 30 18:59:54.218250 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.218197 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-monitoring-tls-certs\") pod \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " Oct 30 18:59:54.218372 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.218254 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-data-science-collector-ta-client-cert\") pod \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\" (UID: \"c23bf958-c5a3-4133-ad82-4b7631a5ae73\") " Oct 30 18:59:54.218571 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.218538 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otc-internal" pod "c23bf958-c5a3-4133-ad82-4b7631a5ae73" (UID: "c23bf958-c5a3-4133-ad82-4b7631a5ae73"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:54.220415 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.220371 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-monitoring-tls-certs" pod "c23bf958-c5a3-4133-ad82-4b7631a5ae73" (UID: "c23bf958-c5a3-4133-ad82-4b7631a5ae73"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:54.220526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.220457 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23bf958-c5a3-4133-ad82-4b7631a5ae73-kube-api-access-q642p" pod "c23bf958-c5a3-4133-ad82-4b7631a5ae73" (UID: "c23bf958-c5a3-4133-ad82-4b7631a5ae73"). InnerVolumeSpecName "kube-api-access-q642p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:54.220526 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.220486 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-data-science-collector-ta-client-cert" pod "c23bf958-c5a3-4133-ad82-4b7631a5ae73" (UID: "c23bf958-c5a3-4133-ad82-4b7631a5ae73"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:54.220796 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.220774 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-tls-certs" pod "c23bf958-c5a3-4133-ad82-4b7631a5ae73" (UID: "c23bf958-c5a3-4133-ad82-4b7631a5ae73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:54.221032 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.221012 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otlp-tls-certs" pod "c23bf958-c5a3-4133-ad82-4b7631a5ae73" (UID: "c23bf958-c5a3-4133-ad82-4b7631a5ae73"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:54.319889 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.319850 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:54.319889 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.319882 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:54.320076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.319896 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:54.320076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.319911 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q642p\" (UniqueName: \"kubernetes.io/projected/c23bf958-c5a3-4133-ad82-4b7631a5ae73-kube-api-access-q642p\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:54.320076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.319925 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23bf958-c5a3-4133-ad82-4b7631a5ae73-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:54.320076 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.319939 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c23bf958-c5a3-4133-ad82-4b7631a5ae73-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:54.780718 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.780682 5014 generic.go:356] "Generic (PLEG): container finished" podID="c23bf958-c5a3-4133-ad82-4b7631a5ae73" containerID="6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f" exitCode=0 Oct 30 18:59:54.780897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.780725 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:54.780897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.780764 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c23bf958-c5a3-4133-ad82-4b7631a5ae73","Type":"ContainerDied","Data":"6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f"} Oct 30 18:59:54.780897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.780826 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c23bf958-c5a3-4133-ad82-4b7631a5ae73","Type":"ContainerDied","Data":"a18a597a765d18619663e9baf8b14309015b11de85baaed834f10b611513bbd3"} Oct 30 18:59:54.780897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.780848 5014 scope.go:122] "RemoveContainer" containerID="6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f" Oct 30 18:59:54.789828 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.789813 5014 scope.go:122] "RemoveContainer" containerID="6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f" Oct 30 18:59:54.790099 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:54.790079 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f\": container with ID starting with 6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f not found: ID does not exist" containerID="6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f" Oct 30 18:59:54.790171 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.790108 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f"} err="failed to get container status \"6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f\": rpc error: code = NotFound desc = could not find container \"6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f\": container with ID starting with 6a6aa2bc4731322ad2b6a815dda82af489cdd79122b0cf57a4db52702a78387f not found: ID does not exist" Oct 30 18:59:54.831460 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.831430 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:54.844064 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.844043 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:54.878933 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.878914 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:54.879291 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.879279 5014 cpu_manager.go:414] "containerMap: removing container" podUID="c23bf958-c5a3-4133-ad82-4b7631a5ae73" containerName="otc-container" Oct 30 18:59:54.879353 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.879292 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23bf958-c5a3-4133-ad82-4b7631a5ae73" containerName="otc-container" Oct 30 18:59:54.879400 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.879356 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23bf958-c5a3-4133-ad82-4b7631a5ae73" containerName="otc-container" Oct 30 18:59:54.883783 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.883767 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:54.901884 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.901864 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-bhcr4\"" Oct 30 18:59:54.912470 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.912448 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:59:54.916193 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.916161 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:54.916469 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:54.916454 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:59:55.001142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.001110 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:59:55.001142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.001113 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:59:55.001142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.001118 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:59:55.025534 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.025511 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.025617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.025543 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vblmm\" (UniqueName: \"kubernetes.io/projected/dcb2e906-299c-4084-8980-c5de4ccf1a8f-kube-api-access-vblmm\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.025617 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.025564 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.025719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.025706 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.025766 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.025754 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.025803 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.025784 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.126297 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.126274 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.126671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.126310 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.126671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.126350 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.126671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.126368 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vblmm\" (UniqueName: \"kubernetes.io/projected/dcb2e906-299c-4084-8980-c5de4ccf1a8f-kube-api-access-vblmm\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.126671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.126390 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.126671 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.126413 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.127355 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.127329 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.129052 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.129026 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.129143 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.129126 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.129249 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.129230 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.129283 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.129226 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.142500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.142480 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23bf958-c5a3-4133-ad82-4b7631a5ae73" path="/var/lib/kubelet/pods/c23bf958-c5a3-4133-ad82-4b7631a5ae73/volumes" Oct 30 18:59:55.149674 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.149656 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vblmm\" (UniqueName: \"kubernetes.io/projected/dcb2e906-299c-4084-8980-c5de4ccf1a8f-kube-api-access-vblmm\") pod \"data-science-collector-collector-0\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.197031 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.197014 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:55.495123 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.495034 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:55.499291 ip-10-0-133-62 kubenswrapper[5014]: W1030 18:59:55.499264 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb2e906_299c_4084_8980_c5de4ccf1a8f.slice/crio-db5abe284b43b471b7a40f9ac68b10ea05e782fa8fe0b53dcebec0d8671e918e WatchSource:0}: Error finding container db5abe284b43b471b7a40f9ac68b10ea05e782fa8fe0b53dcebec0d8671e918e: Status 404 returned error can't find the container with id db5abe284b43b471b7a40f9ac68b10ea05e782fa8fe0b53dcebec0d8671e918e Oct 30 18:59:55.501487 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.501458 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:59:55.501556 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.501512 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:59:55.788049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.787964 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"dcb2e906-299c-4084-8980-c5de4ccf1a8f","Type":"ContainerStarted","Data":"89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf"} Oct 30 18:59:55.788049 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.788018 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"dcb2e906-299c-4084-8980-c5de4ccf1a8f","Type":"ContainerStarted","Data":"db5abe284b43b471b7a40f9ac68b10ea05e782fa8fe0b53dcebec0d8671e918e"} Oct 30 18:59:55.830167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:55.830117 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.830103952 podStartE2EDuration="1.830103952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:55.828894002 +0000 UTC m=+3808.817234488" watchObservedRunningTime="2026-10-30 18:59:55.830103952 +0000 UTC m=+3808.818444439" Oct 30 18:59:58.075489 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.075448 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx"] Oct 30 18:59:58.075969 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.075741 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" podUID="a585f6c7-b0bc-42bd-9cfa-53b7a9d26138" containerName="kube-rbac-proxy" containerID="cri-o://af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499" gracePeriod=30 Oct 30 18:59:58.087121 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.087096 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74"] Oct 30 18:59:58.087312 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.087291 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="kube-rbac-proxy" containerID="cri-o://dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918" gracePeriod=30 Oct 30 18:59:58.087416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.087300 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="method-gate" containerID="cri-o://256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13" gracePeriod=30 Oct 30 18:59:58.087416 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.087300 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="prom-label-proxy" containerID="cri-o://92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165" gracePeriod=30 Oct 30 18:59:58.369194 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.369166 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:59:58.382585 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.382564 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:59:58.459773 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.459738 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g"] Oct 30 18:59:58.459975 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.459949 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" podUID="ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e" containerName="thanos-querier" containerID="cri-o://775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662" gracePeriod=30 Oct 30 18:59:58.461818 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.461789 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-rbac-proxy-config\") pod \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " Oct 30 18:59:58.461974 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.461850 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-client-cert\") pod \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " Oct 30 18:59:58.461974 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.461892 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9207d774-f57c-4091-81a2-f195690744d1-tls\" (UniqueName: \"kubernetes.io/secret/9207d774-f57c-4091-81a2-f195690744d1-tls\") pod \"9207d774-f57c-4091-81a2-f195690744d1\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " Oct 30 18:59:58.461974 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.461930 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-config\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-config\") pod \"9207d774-f57c-4091-81a2-f195690744d1\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " Oct 30 18:59:58.462167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.461983 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-api-access-82lbv\" (UniqueName: \"kubernetes.io/projected/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-api-access-82lbv\") pod \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " Oct 30 18:59:58.462167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.462025 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-tls\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-tls\") pod \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " Oct 30 18:59:58.462167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.462073 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-prometheus-tls-ca\") pod \"9207d774-f57c-4091-81a2-f195690744d1\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " Oct 30 18:59:58.462167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.462102 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-ca\") pod \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\" (UID: \"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138\") " Oct 30 18:59:58.462167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.462131 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-client-ca\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-client-ca\") pod \"9207d774-f57c-4091-81a2-f195690744d1\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " Oct 30 18:59:58.462167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.462164 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9207d774-f57c-4091-81a2-f195690744d1-kube-api-access-rwlct\" (UniqueName: \"kubernetes.io/projected/9207d774-f57c-4091-81a2-f195690744d1-kube-api-access-rwlct\") pod \"9207d774-f57c-4091-81a2-f195690744d1\" (UID: \"9207d774-f57c-4091-81a2-f195690744d1\") " Oct 30 18:59:58.463468 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.463225 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-prometheus-tls-ca" pod "9207d774-f57c-4091-81a2-f195690744d1" (UID: "9207d774-f57c-4091-81a2-f195690744d1"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:58.463795 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.463767 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-ca" pod "a585f6c7-b0bc-42bd-9cfa-53b7a9d26138" (UID: "a585f6c7-b0bc-42bd-9cfa-53b7a9d26138"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:58.464182 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.464134 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-client-ca" pod "9207d774-f57c-4091-81a2-f195690744d1" (UID: "9207d774-f57c-4091-81a2-f195690744d1"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:58.464572 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.464534 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-config" pod "9207d774-f57c-4091-81a2-f195690744d1" (UID: "9207d774-f57c-4091-81a2-f195690744d1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:58.467089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.467000 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-rbac-proxy-config" pod "a585f6c7-b0bc-42bd-9cfa-53b7a9d26138" (UID: "a585f6c7-b0bc-42bd-9cfa-53b7a9d26138"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:58.467089 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.467054 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-api-access-82lbv" pod "a585f6c7-b0bc-42bd-9cfa-53b7a9d26138" (UID: "a585f6c7-b0bc-42bd-9cfa-53b7a9d26138"). InnerVolumeSpecName "kube-api-access-82lbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:58.467377 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.467344 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9207d774-f57c-4091-81a2-f195690744d1-tls" pod "9207d774-f57c-4091-81a2-f195690744d1" (UID: "9207d774-f57c-4091-81a2-f195690744d1"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:58.467460 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.467397 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-tls" pod "a585f6c7-b0bc-42bd-9cfa-53b7a9d26138" (UID: "a585f6c7-b0bc-42bd-9cfa-53b7a9d26138"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:58.467527 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.467497 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9207d774-f57c-4091-81a2-f195690744d1-kube-api-access-rwlct" pod "9207d774-f57c-4091-81a2-f195690744d1" (UID: "9207d774-f57c-4091-81a2-f195690744d1"). InnerVolumeSpecName "kube-api-access-rwlct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:58.468365 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.468037 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-client-cert" pod "a585f6c7-b0bc-42bd-9cfa-53b7a9d26138" (UID: "a585f6c7-b0bc-42bd-9cfa-53b7a9d26138"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:58.563037 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.563006 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.563037 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.563034 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9207d774-f57c-4091-81a2-f195690744d1-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.563306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.563046 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.563306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.563060 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82lbv\" (UniqueName: \"kubernetes.io/projected/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-api-access-82lbv\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.563306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.563071 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.563306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.563079 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-prometheus-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.563306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.563087 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-prometheus-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.563306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.563097 5014 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9207d774-f57c-4091-81a2-f195690744d1-client-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.563306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.563109 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwlct\" (UniqueName: \"kubernetes.io/projected/9207d774-f57c-4091-81a2-f195690744d1-kube-api-access-rwlct\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.563306 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.563120 5014 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138-kube-rbac-proxy-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.661969 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.661937 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:59:58.662488 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.662293 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerName="alertmanager" containerID="cri-o://06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35" gracePeriod=120 Oct 30 18:59:58.662488 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.662319 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerName="config-reloader" containerID="cri-o://16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7" gracePeriod=120 Oct 30 18:59:58.664638 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.664547 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:59:58.665882 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.665402 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="config-reloader" containerID="cri-o://2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c" gracePeriod=600 Oct 30 18:59:58.665882 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.665483 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="thanos-sidecar" containerID="cri-o://025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152" gracePeriod=600 Oct 30 18:59:58.665882 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.665389 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="prometheus" containerID="cri-o://ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8" gracePeriod=600 Oct 30 18:59:58.783761 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.783740 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" Oct 30 18:59:58.803525 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.803496 5014 generic.go:356] "Generic (PLEG): container finished" podID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerID="16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7" exitCode=0 Oct 30 18:59:58.803664 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.803563 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"0282b81d-e88f-4833-8abf-4bac747bce1e","Type":"ContainerDied","Data":"16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7"} Oct 30 18:59:58.804746 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.804724 5014 generic.go:356] "Generic (PLEG): container finished" podID="ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e" containerID="775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662" exitCode=0 Oct 30 18:59:58.804848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.804765 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" Oct 30 18:59:58.804848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.804804 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" event={"ID":"ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e","Type":"ContainerDied","Data":"775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662"} Oct 30 18:59:58.804848 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.804840 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g" event={"ID":"ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e","Type":"ContainerDied","Data":"fcc3dcb6600ce5391b96945e671a95375698c586d3367b78e3899e439f90df9d"} Oct 30 18:59:58.805010 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.804863 5014 scope.go:122] "RemoveContainer" containerID="775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662" Oct 30 18:59:58.806793 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.806774 5014 generic.go:356] "Generic (PLEG): container finished" podID="9207d774-f57c-4091-81a2-f195690744d1" containerID="92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165" exitCode=0 Oct 30 18:59:58.806911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.806795 5014 generic.go:356] "Generic (PLEG): container finished" podID="9207d774-f57c-4091-81a2-f195690744d1" containerID="256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13" exitCode=0 Oct 30 18:59:58.806911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.806805 5014 generic.go:356] "Generic (PLEG): container finished" podID="9207d774-f57c-4091-81a2-f195690744d1" containerID="dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918" exitCode=0 Oct 30 18:59:58.806911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.806819 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" Oct 30 18:59:58.806911 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.806856 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" event={"ID":"9207d774-f57c-4091-81a2-f195690744d1","Type":"ContainerDied","Data":"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165"} Oct 30 18:59:58.807104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.806920 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" event={"ID":"9207d774-f57c-4091-81a2-f195690744d1","Type":"ContainerDied","Data":"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13"} Oct 30 18:59:58.807104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.806962 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" event={"ID":"9207d774-f57c-4091-81a2-f195690744d1","Type":"ContainerDied","Data":"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918"} Oct 30 18:59:58.807104 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.807000 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74" event={"ID":"9207d774-f57c-4091-81a2-f195690744d1","Type":"ContainerDied","Data":"c773c454aef3e07cc35ffb6c3867ced66d898e06a353d8e0368c8ed9875b100e"} Oct 30 18:59:58.809728 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.809706 5014 generic.go:356] "Generic (PLEG): container finished" podID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerID="025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152" exitCode=0 Oct 30 18:59:58.809835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.809729 5014 generic.go:356] "Generic (PLEG): container finished" podID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerID="ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8" exitCode=0 Oct 30 18:59:58.809835 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.809784 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b7b3a82c-4d56-4db5-9918-068bbe69d9d3","Type":"ContainerDied","Data":"025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152"} Oct 30 18:59:58.809943 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.809854 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b7b3a82c-4d56-4db5-9918-068bbe69d9d3","Type":"ContainerDied","Data":"ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8"} Oct 30 18:59:58.811195 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.811174 5014 generic.go:356] "Generic (PLEG): container finished" podID="a585f6c7-b0bc-42bd-9cfa-53b7a9d26138" containerID="af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499" exitCode=0 Oct 30 18:59:58.811266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.811210 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" Oct 30 18:59:58.811266 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.811234 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" event={"ID":"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138","Type":"ContainerDied","Data":"af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499"} Oct 30 18:59:58.811351 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.811270 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx" event={"ID":"a585f6c7-b0bc-42bd-9cfa-53b7a9d26138","Type":"ContainerDied","Data":"e23b94f2e991ea2b2462c0eab18883b6c7d599fece90f7611ead6b499e4bea83"} Oct 30 18:59:58.814202 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.814187 5014 scope.go:122] "RemoveContainer" containerID="775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662" Oct 30 18:59:58.814492 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:58.814467 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662\": container with ID starting with 775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662 not found: ID does not exist" containerID="775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662" Oct 30 18:59:58.814558 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.814504 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662"} err="failed to get container status \"775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662\": rpc error: code = NotFound desc = could not find container \"775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662\": container with ID starting with 775aa9eadaff196b74e3f7aa81d4c91188562498085acb832d7d60f0d90a1662 not found: ID does not exist" Oct 30 18:59:58.814558 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.814545 5014 scope.go:122] "RemoveContainer" containerID="92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165" Oct 30 18:59:58.824067 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.824049 5014 scope.go:122] "RemoveContainer" containerID="256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13" Oct 30 18:59:58.834092 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.834078 5014 scope.go:122] "RemoveContainer" containerID="dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918" Oct 30 18:59:58.841676 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.841657 5014 scope.go:122] "RemoveContainer" containerID="92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165" Oct 30 18:59:58.841925 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:58.841903 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165\": container with ID starting with 92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165 not found: ID does not exist" containerID="92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165" Oct 30 18:59:58.842009 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.841937 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165"} err="failed to get container status \"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165\": rpc error: code = NotFound desc = could not find container \"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165\": container with ID starting with 92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165 not found: ID does not exist" Oct 30 18:59:58.842009 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.841961 5014 scope.go:122] "RemoveContainer" containerID="256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13" Oct 30 18:59:58.842232 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:58.842214 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13\": container with ID starting with 256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13 not found: ID does not exist" containerID="256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13" Oct 30 18:59:58.842310 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.842236 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13"} err="failed to get container status \"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13\": rpc error: code = NotFound desc = could not find container \"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13\": container with ID starting with 256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13 not found: ID does not exist" Oct 30 18:59:58.842310 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.842258 5014 scope.go:122] "RemoveContainer" containerID="dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918" Oct 30 18:59:58.842526 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:58.842497 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918\": container with ID starting with dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918 not found: ID does not exist" containerID="dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918" Oct 30 18:59:58.842620 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.842531 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918"} err="failed to get container status \"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918\": rpc error: code = NotFound desc = could not find container \"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918\": container with ID starting with dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918 not found: ID does not exist" Oct 30 18:59:58.842620 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.842549 5014 scope.go:122] "RemoveContainer" containerID="92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165" Oct 30 18:59:58.842798 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.842776 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165"} err="failed to get container status \"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165\": rpc error: code = NotFound desc = could not find container \"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165\": container with ID starting with 92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165 not found: ID does not exist" Oct 30 18:59:58.842798 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.842797 5014 scope.go:122] "RemoveContainer" containerID="256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13" Oct 30 18:59:58.843023 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.843004 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13"} err="failed to get container status \"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13\": rpc error: code = NotFound desc = could not find container \"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13\": container with ID starting with 256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13 not found: ID does not exist" Oct 30 18:59:58.843101 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.843023 5014 scope.go:122] "RemoveContainer" containerID="dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918" Oct 30 18:59:58.843216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.843200 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918"} err="failed to get container status \"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918\": rpc error: code = NotFound desc = could not find container \"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918\": container with ID starting with dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918 not found: ID does not exist" Oct 30 18:59:58.843216 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.843215 5014 scope.go:122] "RemoveContainer" containerID="92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165" Oct 30 18:59:58.843412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.843394 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165"} err="failed to get container status \"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165\": rpc error: code = NotFound desc = could not find container \"92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165\": container with ID starting with 92e5e2bfc77ae93ffb288860d2d0e51c0dfc4905a3d5844e72c2bc865e53e165 not found: ID does not exist" Oct 30 18:59:58.843412 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.843412 5014 scope.go:122] "RemoveContainer" containerID="256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13" Oct 30 18:59:58.843622 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.843586 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13"} err="failed to get container status \"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13\": rpc error: code = NotFound desc = could not find container \"256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13\": container with ID starting with 256c9fe326a1808476b604f6e493f6d027852e27436087a936ff92b81fe85a13 not found: ID does not exist" Oct 30 18:59:58.843706 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.843622 5014 scope.go:122] "RemoveContainer" containerID="dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918" Oct 30 18:59:58.843824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.843807 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918"} err="failed to get container status \"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918\": rpc error: code = NotFound desc = could not find container \"dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918\": container with ID starting with dcd8b5b952e0aef773942c7c17d512caa5d82fcdd153a1ca4125be04dd3b1918 not found: ID does not exist" Oct 30 18:59:58.843824 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.843823 5014 scope.go:122] "RemoveContainer" containerID="af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499" Oct 30 18:59:58.852065 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.852052 5014 scope.go:122] "RemoveContainer" containerID="af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499" Oct 30 18:59:58.852297 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:58.852278 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499\": container with ID starting with af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499 not found: ID does not exist" containerID="af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499" Oct 30 18:59:58.852419 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.852301 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499"} err="failed to get container status \"af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499\": rpc error: code = NotFound desc = could not find container \"af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499\": container with ID starting with af94b88f9e7ae814b536c8b793db0d7111d43a8bde036cb027ab57657e1a9499 not found: ID does not exist" Oct 30 18:59:58.865454 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.865434 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e-kube-api-access-hscn8\" (UniqueName: \"kubernetes.io/projected/ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e-kube-api-access-hscn8\") pod \"ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e\" (UID: \"ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e\") " Oct 30 18:59:58.867403 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.867385 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e-kube-api-access-hscn8" pod "ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e" (UID: "ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e"). InnerVolumeSpecName "kube-api-access-hscn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:58.884318 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.884290 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74"] Oct 30 18:59:58.894293 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.894274 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-nbk74"] Oct 30 18:59:58.951191 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.951167 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx"] Oct 30 18:59:58.956039 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.956020 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7lswx"] Oct 30 18:59:58.966341 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:58.966322 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hscn8\" (UniqueName: \"kubernetes.io/projected/ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e-kube-api-access-hscn8\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.141760 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.141708 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9207d774-f57c-4091-81a2-f195690744d1" path="/var/lib/kubelet/pods/9207d774-f57c-4091-81a2-f195690744d1/volumes" Oct 30 18:59:59.142223 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.142209 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a585f6c7-b0bc-42bd-9cfa-53b7a9d26138" path="/var/lib/kubelet/pods/a585f6c7-b0bc-42bd-9cfa-53b7a9d26138/volumes" Oct 30 18:59:59.147948 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.147929 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g"] Oct 30 18:59:59.154648 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.154629 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5s66g"] Oct 30 18:59:59.353666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.352777 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:59.353666 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.353214 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="dcb2e906-299c-4084-8980-c5de4ccf1a8f" containerName="otc-container" containerID="cri-o://89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf" gracePeriod=30 Oct 30 18:59:59.503169 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.502820 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572014 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-kube-api-access-q8z4r\" (UniqueName: \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-kube-api-access-q8z4r\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572064 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572117 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-tls-assets\" (UniqueName: \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-tls-assets\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572143 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572185 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572217 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config-out\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572344 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572397 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572435 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572462 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-thanos-prometheus-http-client-file\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572513 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.572571 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-secret-prometheus-web-tls-ca\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.573227 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\" (UID: \"b7b3a82c-4d56-4db5-9918-068bbe69d9d3\") " Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.575132 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-0" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:59.575543 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.575464 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-1" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:59.576519 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.575822 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.576519 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.575919 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-2" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:59.577787 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.577753 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config-out" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:59.578960 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.578934 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-tls-assets" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:59.579560 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.579536 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-kube-api-access-q8z4r" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "kube-api-access-q8z4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:59.581251 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.581229 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.581354 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.581248 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-thanos-prometheus-http-client-file" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.583083 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.583060 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-secret-prometheus-web-tls-ca" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.583167 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.583086 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.585634 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.585615 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "pvc-c02e8472-bb1a-4974-aa89-86eef715889b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:59:59.597208 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.597151 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config" pod "b7b3a82c-4d56-4db5-9918-068bbe69d9d3" (UID: "b7b3a82c-4d56-4db5-9918-068bbe69d9d3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.608719 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.608699 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:59.646041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.646020 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:59.673980 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.673954 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-kube-api-access-vg6zh\" (UniqueName: \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-kube-api-access-vg6zh\") pod \"0282b81d-e88f-4833-8abf-4bac747bce1e\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " Oct 30 18:59:59.674142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.673989 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-config-out\") pod \"0282b81d-e88f-4833-8abf-4bac747bce1e\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " Oct 30 18:59:59.674142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674024 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-web-config\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-web-config\") pod \"0282b81d-e88f-4833-8abf-4bac747bce1e\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " Oct 30 18:59:59.674142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674045 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-data-science-collector-ta-client-cert\") pod \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " Oct 30 18:59:59.674142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674064 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-cluster-tls-config\") pod \"0282b81d-e88f-4833-8abf-4bac747bce1e\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " Oct 30 18:59:59.674142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674088 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-monitoring-tls-certs\") pod \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " Oct 30 18:59:59.674142 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674119 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-tls-certs\") pod \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " Oct 30 18:59:59.674479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674186 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb2e906-299c-4084-8980-c5de4ccf1a8f-kube-api-access-vblmm\" (UniqueName: \"kubernetes.io/projected/dcb2e906-299c-4084-8980-c5de4ccf1a8f-kube-api-access-vblmm\") pod \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " Oct 30 18:59:59.674479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674222 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otc-internal\") pod \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " Oct 30 18:59:59.674479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674276 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-tls-assets\" (UniqueName: \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-tls-assets\") pod \"0282b81d-e88f-4833-8abf-4bac747bce1e\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " Oct 30 18:59:59.674479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674308 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otlp-tls-certs\") pod \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\" (UID: \"dcb2e906-299c-4084-8980-c5de4ccf1a8f\") " Oct 30 18:59:59.674479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674351 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-config-volume\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-config-volume\") pod \"0282b81d-e88f-4833-8abf-4bac747bce1e\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " Oct 30 18:59:59.674479 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674386 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-alertmanager-data-science-monitoringstack-db\") pod \"0282b81d-e88f-4833-8abf-4bac747bce1e\" (UID: \"0282b81d-e88f-4833-8abf-4bac747bce1e\") " Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674690 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674715 5014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674731 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674747 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8z4r\" (UniqueName: \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-kube-api-access-q8z4r\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674762 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674777 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674790 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674805 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674820 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674850 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674862 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674876 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675001 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.674891 5014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7b3a82c-4d56-4db5-9918-068bbe69d9d3-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.675708 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.675586 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-alertmanager-data-science-monitoringstack-db" pod "0282b81d-e88f-4833-8abf-4bac747bce1e" (UID: "0282b81d-e88f-4833-8abf-4bac747bce1e"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:59.675770 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.675725 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otc-internal" pod "dcb2e906-299c-4084-8980-c5de4ccf1a8f" (UID: "dcb2e906-299c-4084-8980-c5de4ccf1a8f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:59.676987 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.676948 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-data-science-collector-ta-client-cert" pod "dcb2e906-299c-4084-8980-c5de4ccf1a8f" (UID: "dcb2e906-299c-4084-8980-c5de4ccf1a8f"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.678041 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.677897 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-kube-api-access-vg6zh" pod "0282b81d-e88f-4833-8abf-4bac747bce1e" (UID: "0282b81d-e88f-4833-8abf-4bac747bce1e"). InnerVolumeSpecName "kube-api-access-vg6zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:59.679430 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.679404 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-tls-certs" pod "dcb2e906-299c-4084-8980-c5de4ccf1a8f" (UID: "dcb2e906-299c-4084-8980-c5de4ccf1a8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.679657 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.679472 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-tls-assets" pod "0282b81d-e88f-4833-8abf-4bac747bce1e" (UID: "0282b81d-e88f-4833-8abf-4bac747bce1e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:59.679906 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.679879 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-config-volume" pod "0282b81d-e88f-4833-8abf-4bac747bce1e" (UID: "0282b81d-e88f-4833-8abf-4bac747bce1e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.680153 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.680125 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-monitoring-tls-certs" pod "dcb2e906-299c-4084-8980-c5de4ccf1a8f" (UID: "dcb2e906-299c-4084-8980-c5de4ccf1a8f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.680153 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.680135 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-config-out" pod "0282b81d-e88f-4833-8abf-4bac747bce1e" (UID: "0282b81d-e88f-4833-8abf-4bac747bce1e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:59.680907 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.680875 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otlp-tls-certs" pod "dcb2e906-299c-4084-8980-c5de4ccf1a8f" (UID: "dcb2e906-299c-4084-8980-c5de4ccf1a8f"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.683046 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.683021 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb2e906-299c-4084-8980-c5de4ccf1a8f-kube-api-access-vblmm" pod "dcb2e906-299c-4084-8980-c5de4ccf1a8f" (UID: "dcb2e906-299c-4084-8980-c5de4ccf1a8f"). InnerVolumeSpecName "kube-api-access-vblmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:59.685441 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.685414 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-cluster-tls-config" pod "0282b81d-e88f-4833-8abf-4bac747bce1e" (UID: "0282b81d-e88f-4833-8abf-4bac747bce1e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.689341 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.689322 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c02e8472-bb1a-4974-aa89-86eef715889b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e") on node "ip-10-0-133-62.ec2.internal" Oct 30 18:59:59.693456 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.693435 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-web-config" pod "0282b81d-e88f-4833-8abf-4bac747bce1e" (UID: "0282b81d-e88f-4833-8abf-4bac747bce1e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.775500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775403 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775436 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775451 5014 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-cluster-tls-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775466 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775479 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775491 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vblmm\" (UniqueName: \"kubernetes.io/projected/dcb2e906-299c-4084-8980-c5de4ccf1a8f-kube-api-access-vblmm\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775500 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775507 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" Oct 30 18:59:59.775897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775522 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775533 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775544 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb2e906-299c-4084-8980-c5de4ccf1a8f-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775556 5014 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0282b81d-e88f-4833-8abf-4bac747bce1e-config-volume\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775568 5014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775583 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6zh\" (UniqueName: \"kubernetes.io/projected/0282b81d-e88f-4833-8abf-4bac747bce1e-kube-api-access-vg6zh\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.775897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.775624 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0282b81d-e88f-4833-8abf-4bac747bce1e-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.826651 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.826586 5014 generic.go:356] "Generic (PLEG): container finished" podID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerID="06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35" exitCode=0 Oct 30 18:59:59.826651 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.826627 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"0282b81d-e88f-4833-8abf-4bac747bce1e","Type":"ContainerDied","Data":"06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35"} Oct 30 18:59:59.826897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.826662 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:59:59.826897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.826717 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"0282b81d-e88f-4833-8abf-4bac747bce1e","Type":"ContainerDied","Data":"e5bdcb41e61ed843a09b62edae3d1569be9485655aa986698f8cc9a3d1e52fa8"} Oct 30 18:59:59.826897 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.826744 5014 scope.go:122] "RemoveContainer" containerID="16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7" Oct 30 18:59:59.831011 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.830978 5014 generic.go:356] "Generic (PLEG): container finished" podID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerID="2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c" exitCode=0 Oct 30 18:59:59.831120 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.831060 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b7b3a82c-4d56-4db5-9918-068bbe69d9d3","Type":"ContainerDied","Data":"2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c"} Oct 30 18:59:59.831192 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.831125 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b7b3a82c-4d56-4db5-9918-068bbe69d9d3","Type":"ContainerDied","Data":"4c47b393e42502a55b3b8da83e5ca3aebda50434580bdff6f3870cc119aebde8"} Oct 30 18:59:59.831363 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.831344 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:59:59.832456 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.832434 5014 generic.go:356] "Generic (PLEG): container finished" podID="dcb2e906-299c-4084-8980-c5de4ccf1a8f" containerID="89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf" exitCode=0 Oct 30 18:59:59.832571 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.832494 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"dcb2e906-299c-4084-8980-c5de4ccf1a8f","Type":"ContainerDied","Data":"89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf"} Oct 30 18:59:59.832571 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.832520 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"dcb2e906-299c-4084-8980-c5de4ccf1a8f","Type":"ContainerDied","Data":"db5abe284b43b471b7a40f9ac68b10ea05e782fa8fe0b53dcebec0d8671e918e"} Oct 30 18:59:59.832571 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.832571 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:59:59.836826 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.836697 5014 scope.go:122] "RemoveContainer" containerID="06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35" Oct 30 18:59:59.844995 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.844982 5014 scope.go:122] "RemoveContainer" containerID="c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3" Oct 30 18:59:59.853662 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.853642 5014 scope.go:122] "RemoveContainer" containerID="16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7" Oct 30 18:59:59.853948 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:59.853926 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7\": container with ID starting with 16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7 not found: ID does not exist" containerID="16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7" Oct 30 18:59:59.854009 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.853961 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7"} err="failed to get container status \"16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7\": rpc error: code = NotFound desc = could not find container \"16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7\": container with ID starting with 16d4f09bbd5bd9f9b6b544696b122fd853c058f457cd97b2f8b52d0d88497bf7 not found: ID does not exist" Oct 30 18:59:59.854009 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.853984 5014 scope.go:122] "RemoveContainer" containerID="06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35" Oct 30 18:59:59.854222 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:59.854205 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35\": container with ID starting with 06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35 not found: ID does not exist" containerID="06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35" Oct 30 18:59:59.854288 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.854231 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35"} err="failed to get container status \"06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35\": rpc error: code = NotFound desc = could not find container \"06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35\": container with ID starting with 06addb62c63ea02cc487c59163e3399f6c96c06404133a3df87e878769c93d35 not found: ID does not exist" Oct 30 18:59:59.854288 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.854253 5014 scope.go:122] "RemoveContainer" containerID="c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3" Oct 30 18:59:59.854496 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:59.854477 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3\": container with ID starting with c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3 not found: ID does not exist" containerID="c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3" Oct 30 18:59:59.854535 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.854506 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3"} err="failed to get container status \"c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3\": rpc error: code = NotFound desc = could not find container \"c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3\": container with ID starting with c617d011900b21dd3a01eb302ca50dcb4f8bb2b7b9a89553242aaebd88a602f3 not found: ID does not exist" Oct 30 18:59:59.854535 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.854527 5014 scope.go:122] "RemoveContainer" containerID="025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152" Oct 30 18:59:59.863027 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.863005 5014 scope.go:122] "RemoveContainer" containerID="2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c" Oct 30 18:59:59.871652 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.871635 5014 scope.go:122] "RemoveContainer" containerID="ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8" Oct 30 18:59:59.879317 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.879299 5014 scope.go:122] "RemoveContainer" containerID="5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4" Oct 30 18:59:59.886146 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.886126 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:59.887422 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.887408 5014 scope.go:122] "RemoveContainer" containerID="025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152" Oct 30 18:59:59.887663 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:59.887642 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152\": container with ID starting with 025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152 not found: ID does not exist" containerID="025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152" Oct 30 18:59:59.887753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.887671 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152"} err="failed to get container status \"025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152\": rpc error: code = NotFound desc = could not find container \"025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152\": container with ID starting with 025e868ea42add69bf79ec21c5bec98d7298846ff092824ebcfab07e3c84c152 not found: ID does not exist" Oct 30 18:59:59.887753 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.887693 5014 scope.go:122] "RemoveContainer" containerID="2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c" Oct 30 18:59:59.887943 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:59.887927 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c\": container with ID starting with 2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c not found: ID does not exist" containerID="2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c" Oct 30 18:59:59.887989 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.887957 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c"} err="failed to get container status \"2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c\": rpc error: code = NotFound desc = could not find container \"2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c\": container with ID starting with 2c8aa123079b3bda8287aac93f6e3ed6f6480ee7ed344f4546bdc37dc707b02c not found: ID does not exist" Oct 30 18:59:59.887989 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.887971 5014 scope.go:122] "RemoveContainer" containerID="ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8" Oct 30 18:59:59.888177 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:59.888161 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8\": container with ID starting with ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8 not found: ID does not exist" containerID="ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8" Oct 30 18:59:59.888220 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.888181 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8"} err="failed to get container status \"ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8\": rpc error: code = NotFound desc = could not find container \"ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8\": container with ID starting with ab8382e62f7549d1fdccdb7ca81acfd150ae9f796daf360f46587c0d9354f5b8 not found: ID does not exist" Oct 30 18:59:59.888220 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.888202 5014 scope.go:122] "RemoveContainer" containerID="5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4" Oct 30 18:59:59.888391 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:59.888373 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4\": container with ID starting with 5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4 not found: ID does not exist" containerID="5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4" Oct 30 18:59:59.888433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.888398 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4"} err="failed to get container status \"5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4\": rpc error: code = NotFound desc = could not find container \"5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4\": container with ID starting with 5afba7ebd313ae9218569b83b20be1382f42301f958680a5766711590fa1ebc4 not found: ID does not exist" Oct 30 18:59:59.888433 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.888413 5014 scope.go:122] "RemoveContainer" containerID="89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf" Oct 30 18:59:59.893767 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.893746 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:59:59.895907 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.895892 5014 scope.go:122] "RemoveContainer" containerID="89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf" Oct 30 18:59:59.896194 ip-10-0-133-62 kubenswrapper[5014]: E1030 18:59:59.896173 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf\": container with ID starting with 89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf not found: ID does not exist" containerID="89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf" Oct 30 18:59:59.896268 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.896199 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf"} err="failed to get container status \"89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf\": rpc error: code = NotFound desc = could not find container \"89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf\": container with ID starting with 89b2cb668d14eb73e39f19df7fef407d099a36b8addfdffd4f72d8fca584acdf not found: ID does not exist" Oct 30 18:59:59.933215 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.933186 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:59:59.938193 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.938174 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:59:59.975357 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.975334 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:59:59.983474 ip-10-0-133-62 kubenswrapper[5014]: I1030 18:59:59.983448 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 19:00:01.141539 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:01.141497 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0282b81d-e88f-4833-8abf-4bac747bce1e" path="/var/lib/kubelet/pods/0282b81d-e88f-4833-8abf-4bac747bce1e/volumes" Oct 30 19:00:01.142011 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:01.141969 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e" path="/var/lib/kubelet/pods/ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e/volumes" Oct 30 19:00:01.142402 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:01.142385 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" path="/var/lib/kubelet/pods/b7b3a82c-4d56-4db5-9918-068bbe69d9d3/volumes" Oct 30 19:00:01.143270 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:01.143251 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb2e906-299c-4084-8980-c5de4ccf1a8f" path="/var/lib/kubelet/pods/dcb2e906-299c-4084-8980-c5de4ccf1a8f/volumes" Oct 30 19:00:11.636437 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636354 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl"] Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636766 5014 cpu_manager.go:414] "containerMap: removing container" podUID="a585f6c7-b0bc-42bd-9cfa-53b7a9d26138" containerName="kube-rbac-proxy" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636783 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a585f6c7-b0bc-42bd-9cfa-53b7a9d26138" containerName="kube-rbac-proxy" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636801 5014 cpu_manager.go:414] "containerMap: removing container" podUID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerName="config-reloader" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636809 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerName="config-reloader" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636820 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="init-config-reloader" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636827 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="init-config-reloader" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636844 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="config-reloader" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636851 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="config-reloader" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636866 5014 cpu_manager.go:414] "containerMap: removing container" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="method-gate" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636873 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="method-gate" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636883 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="thanos-sidecar" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636891 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="thanos-sidecar" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636907 5014 cpu_manager.go:414] "containerMap: removing container" podUID="ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e" containerName="thanos-querier" Oct 30 19:00:11.636906 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636914 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e" containerName="thanos-querier" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636932 5014 cpu_manager.go:414] "containerMap: removing container" podUID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerName="init-config-reloader" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636937 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerName="init-config-reloader" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636953 5014 cpu_manager.go:414] "containerMap: removing container" podUID="dcb2e906-299c-4084-8980-c5de4ccf1a8f" containerName="otc-container" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636961 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb2e906-299c-4084-8980-c5de4ccf1a8f" containerName="otc-container" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636973 5014 cpu_manager.go:414] "containerMap: removing container" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="kube-rbac-proxy" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636980 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="kube-rbac-proxy" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636992 5014 cpu_manager.go:414] "containerMap: removing container" podUID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerName="alertmanager" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.636998 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerName="alertmanager" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637007 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="prometheus" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637014 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="prometheus" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637030 5014 cpu_manager.go:414] "containerMap: removing container" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="prom-label-proxy" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637035 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="prom-label-proxy" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637119 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="prom-label-proxy" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637127 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="a585f6c7-b0bc-42bd-9cfa-53b7a9d26138" containerName="kube-rbac-proxy" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637133 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerName="config-reloader" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637140 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="config-reloader" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637147 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="method-gate" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637154 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="thanos-sidecar" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637160 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5c7eee-8b8f-4e8a-81c1-3f291319ac8e" containerName="thanos-querier" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637167 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb2e906-299c-4084-8980-c5de4ccf1a8f" containerName="otc-container" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637172 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="9207d774-f57c-4091-81a2-f195690744d1" containerName="kube-rbac-proxy" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637178 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0282b81d-e88f-4833-8abf-4bac747bce1e" containerName="alertmanager" Oct 30 19:00:11.637534 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.637182 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b3a82c-4d56-4db5-9918-068bbe69d9d3" containerName="prometheus" Oct 30 19:00:11.640417 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.640401 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.658476 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.658447 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.658584 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.658490 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.658584 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.658554 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9z9h\" (UniqueName: \"kubernetes.io/projected/0b7e0f15-c382-45ba-9684-5b0074548309-kube-api-access-x9z9h\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.658742 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.658725 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.658821 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.658736 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:00:11.658821 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.658762 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.666444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.666425 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 19:00:11.666803 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.666785 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 19:00:11.666803 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.666797 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-55j6n\"" Oct 30 19:00:11.668392 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.668372 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl"] Oct 30 19:00:11.678259 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.678239 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5"] Oct 30 19:00:11.682089 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.682071 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.704243 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.704217 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5"] Oct 30 19:00:11.755096 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.755064 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 19:00:11.755096 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.755086 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 19:00:11.755096 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.755098 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-xv8lr\"" Oct 30 19:00:11.755344 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.755130 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 19:00:11.755344 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.755142 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 19:00:11.755344 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.755149 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 19:00:11.755344 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.755183 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:00:11.759579 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.759557 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7f5f\" (UniqueName: \"kubernetes.io/projected/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-kube-api-access-d7f5f\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.759710 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.759624 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.759775 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.759702 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.759775 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.759739 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.759775 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.759762 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.759941 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.759820 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.759941 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.759865 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.759941 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.759910 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.760101 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.759946 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9z9h\" (UniqueName: \"kubernetes.io/projected/0b7e0f15-c382-45ba-9684-5b0074548309-kube-api-access-x9z9h\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.760101 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.760091 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.760654 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.760631 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.762387 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.762369 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.762931 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.762911 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.763108 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.763086 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.787282 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.787247 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9z9h\" (UniqueName: \"kubernetes.io/projected/0b7e0f15-c382-45ba-9684-5b0074548309-kube-api-access-x9z9h\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2vkpl\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.860883 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.860859 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.860992 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.860893 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.860992 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.860915 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7f5f\" (UniqueName: \"kubernetes.io/projected/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-kube-api-access-d7f5f\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.860992 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.860936 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.860992 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.860966 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.861530 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.861510 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.861615 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.861538 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.861655 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.861632 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.863226 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.863205 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.884073 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.884043 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7f5f\" (UniqueName: \"kubernetes.io/projected/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-kube-api-access-d7f5f\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:11.952570 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.952511 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:11.998786 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:11.998754 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:12.248003 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:12.247926 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl"] Oct 30 19:00:12.252107 ip-10-0-133-62 kubenswrapper[5014]: W1030 19:00:12.252079 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7e0f15_c382_45ba_9684_5b0074548309.slice/crio-e3b68516c6bcea75607531ff8f4e5b5f20a5477869a0f22dcdbfe8bf04f8a74c WatchSource:0}: Error finding container e3b68516c6bcea75607531ff8f4e5b5f20a5477869a0f22dcdbfe8bf04f8a74c: Status 404 returned error can't find the container with id e3b68516c6bcea75607531ff8f4e5b5f20a5477869a0f22dcdbfe8bf04f8a74c Oct 30 19:00:12.300352 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:12.300312 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5"] Oct 30 19:00:12.303956 ip-10-0-133-62 kubenswrapper[5014]: W1030 19:00:12.303930 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102f1fe2_9102_426c_9c6c_1c63c7e7dad8.slice/crio-5a419ce8e30740649bee5d0461562ac7cf8305866ceab6a3c3f6c8d54731b34d WatchSource:0}: Error finding container 5a419ce8e30740649bee5d0461562ac7cf8305866ceab6a3c3f6c8d54731b34d: Status 404 returned error can't find the container with id 5a419ce8e30740649bee5d0461562ac7cf8305866ceab6a3c3f6c8d54731b34d Oct 30 19:00:12.908387 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:12.908351 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" event={"ID":"0b7e0f15-c382-45ba-9684-5b0074548309","Type":"ContainerStarted","Data":"fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd"} Oct 30 19:00:12.908778 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:12.908441 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:12.908778 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:12.908469 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" event={"ID":"0b7e0f15-c382-45ba-9684-5b0074548309","Type":"ContainerStarted","Data":"e3b68516c6bcea75607531ff8f4e5b5f20a5477869a0f22dcdbfe8bf04f8a74c"} Oct 30 19:00:12.910528 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:12.910507 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" event={"ID":"102f1fe2-9102-426c-9c6c-1c63c7e7dad8","Type":"ContainerStarted","Data":"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c"} Oct 30 19:00:12.910638 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:12.910555 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" event={"ID":"102f1fe2-9102-426c-9c6c-1c63c7e7dad8","Type":"ContainerStarted","Data":"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe"} Oct 30 19:00:12.910638 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:12.910578 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" event={"ID":"102f1fe2-9102-426c-9c6c-1c63c7e7dad8","Type":"ContainerStarted","Data":"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b"} Oct 30 19:00:12.910638 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:12.910629 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" event={"ID":"102f1fe2-9102-426c-9c6c-1c63c7e7dad8","Type":"ContainerStarted","Data":"5a419ce8e30740649bee5d0461562ac7cf8305866ceab6a3c3f6c8d54731b34d"} Oct 30 19:00:12.948584 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:12.948536 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" podStartSLOduration=1.9485244499999999 podStartE2EDuration="1.94852445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:12.94798725 +0000 UTC m=+3825.936327742" watchObservedRunningTime="2026-10-30 19:00:12.94852445 +0000 UTC m=+3825.936864936" Oct 30 19:00:12.994791 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:12.994753 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" podStartSLOduration=1.9947417349999998 podStartE2EDuration="1.994741735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:12.992392727 +0000 UTC m=+3825.980733214" watchObservedRunningTime="2026-10-30 19:00:12.994741735 +0000 UTC m=+3825.983082237" Oct 30 19:00:13.164287 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.164212 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 19:00:13.168030 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.168007 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.183291 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.183245 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 19:00:13.183397 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.183386 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 19:00:13.183496 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.183483 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 19:00:13.183626 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.183612 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-npstw\"" Oct 30 19:00:13.183786 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.183772 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 19:00:13.184967 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.184948 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 19:00:13.195496 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.195473 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 19:00:13.274775 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.274742 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/99afabe9-bc71-4663-a085-70939405f3dd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.274967 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.274785 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.274967 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.274882 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.274967 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.274927 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.275158 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.275051 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55fdx\" (UniqueName: \"kubernetes.io/projected/99afabe9-bc71-4663-a085-70939405f3dd-kube-api-access-55fdx\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.275158 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.275134 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.333834 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:00:13.333793 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 19:00:13.375526 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.375498 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.375701 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.375540 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/99afabe9-bc71-4663-a085-70939405f3dd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.375701 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.375558 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.375701 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.375617 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.375701 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.375645 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.375701 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.375682 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55fdx\" (UniqueName: \"kubernetes.io/projected/99afabe9-bc71-4663-a085-70939405f3dd-kube-api-access-55fdx\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.376703 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.376672 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/99afabe9-bc71-4663-a085-70939405f3dd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.378262 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.378237 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.378384 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.378310 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.378384 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.378364 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.378547 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.378523 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.398683 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.398662 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55fdx\" (UniqueName: \"kubernetes.io/projected/99afabe9-bc71-4663-a085-70939405f3dd-kube-api-access-55fdx\") pod \"data-science-collector-collector-0\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.484757 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.484697 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:13.784933 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.784835 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 19:00:13.789143 ip-10-0-133-62 kubenswrapper[5014]: W1030 19:00:13.789116 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99afabe9_bc71_4663_a085_70939405f3dd.slice/crio-4fa04489f3e785ca9beea95d98d1bfecf867aef0a43f27ce11552b5a3d62b4d6 WatchSource:0}: Error finding container 4fa04489f3e785ca9beea95d98d1bfecf867aef0a43f27ce11552b5a3d62b4d6: Status 404 returned error can't find the container with id 4fa04489f3e785ca9beea95d98d1bfecf867aef0a43f27ce11552b5a3d62b4d6 Oct 30 19:00:13.791576 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.791549 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:00:13.791664 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.791646 5014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:00:13.918132 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.918095 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"99afabe9-bc71-4663-a085-70939405f3dd","Type":"ContainerStarted","Data":"ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c"} Oct 30 19:00:13.918589 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.918160 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"99afabe9-bc71-4663-a085-70939405f3dd","Type":"ContainerStarted","Data":"4fa04489f3e785ca9beea95d98d1bfecf867aef0a43f27ce11552b5a3d62b4d6"} Oct 30 19:00:13.960225 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:13.960171 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=0.960156923 podStartE2EDuration="960.156923ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:13.958987446 +0000 UTC m=+3826.947327933" watchObservedRunningTime="2026-10-30 19:00:13.960156923 +0000 UTC m=+3826.948497409" Oct 30 19:00:18.920788 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:18.920755 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:20.149742 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.149706 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst"] Oct 30 19:00:20.153370 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.153346 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" Oct 30 19:00:20.172441 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.172416 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-5mtd6\"" Oct 30 19:00:20.185368 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.185334 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst"] Oct 30 19:00:20.228616 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.228574 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4455l\" (UniqueName: \"kubernetes.io/projected/4845506f-722e-4a85-abbb-f3bb3801d001-kube-api-access-4455l\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst\" (UID: \"4845506f-722e-4a85-abbb-f3bb3801d001\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" Oct 30 19:00:20.329986 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.329955 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4455l\" (UniqueName: \"kubernetes.io/projected/4845506f-722e-4a85-abbb-f3bb3801d001-kube-api-access-4455l\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst\" (UID: \"4845506f-722e-4a85-abbb-f3bb3801d001\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" Oct 30 19:00:20.353325 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.353292 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4455l\" (UniqueName: \"kubernetes.io/projected/4845506f-722e-4a85-abbb-f3bb3801d001-kube-api-access-4455l\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst\" (UID: \"4845506f-722e-4a85-abbb-f3bb3801d001\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" Oct 30 19:00:20.474485 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.474421 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" Oct 30 19:00:20.769745 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.769667 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst"] Oct 30 19:00:20.776893 ip-10-0-133-62 kubenswrapper[5014]: W1030 19:00:20.776865 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4845506f_722e_4a85_abbb_f3bb3801d001.slice/crio-526fad875e589ed1734e8a22f3d1462fe8e1008208cd6af56285df8542592dc7 WatchSource:0}: Error finding container 526fad875e589ed1734e8a22f3d1462fe8e1008208cd6af56285df8542592dc7: Status 404 returned error can't find the container with id 526fad875e589ed1734e8a22f3d1462fe8e1008208cd6af56285df8542592dc7 Oct 30 19:00:20.955123 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.955091 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" event={"ID":"4845506f-722e-4a85-abbb-f3bb3801d001","Type":"ContainerStarted","Data":"dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c"} Oct 30 19:00:20.955123 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.955127 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" event={"ID":"4845506f-722e-4a85-abbb-f3bb3801d001","Type":"ContainerStarted","Data":"526fad875e589ed1734e8a22f3d1462fe8e1008208cd6af56285df8542592dc7"} Oct 30 19:00:20.993325 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:20.993280 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" podStartSLOduration=0.993266428 podStartE2EDuration="993.266428ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:20.992526081 +0000 UTC m=+3833.980866568" watchObservedRunningTime="2026-10-30 19:00:20.993266428 +0000 UTC m=+3833.981606915" Oct 30 19:00:21.039120 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.039047 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 19:00:21.043978 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.043540 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.074267 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.074242 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 19:00:21.105476 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.105452 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 19:00:21.109702 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.109682 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.129009 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.128987 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 19:00:21.129131 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.129054 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 19:00:21.129131 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.129067 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 19:00:21.129131 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.129091 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 19:00:21.129131 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.129110 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 19:00:21.129131 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.129132 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 19:00:21.129467 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.129147 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-72b7s\"" Oct 30 19:00:21.129467 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.129170 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 19:00:21.129467 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.129196 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 19:00:21.129467 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.129197 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 19:00:21.129467 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.129240 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-hg6c9\"" Oct 30 19:00:21.136029 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.136000 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.136124 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.136075 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.136124 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.136109 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.136397 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.136135 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.136397 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.136207 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdgp\" (UniqueName: \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-kube-api-access-4jdgp\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.136397 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.136238 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.136397 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.136269 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.142318 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.142296 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 19:00:21.199505 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.199483 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 19:00:21.199859 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.199574 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 19:00:21.199859 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.199764 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 19:00:21.237107 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237085 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj74n\" (UniqueName: \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-kube-api-access-cj74n\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.237200 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237119 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.237200 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237143 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.237283 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237231 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.237329 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237295 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.237329 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237319 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.237407 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237335 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.237407 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237362 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7492ab70-7728-4bdf-ae23-9842106016fd-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.237482 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237464 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.237521 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237502 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdgp\" (UniqueName: \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-kube-api-access-4jdgp\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.237623 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237581 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.237761 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237739 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.237835 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237747 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.237835 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237801 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.237939 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237835 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.237939 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237866 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.238051 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237959 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.238051 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.237995 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.238051 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.238043 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.238187 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.238109 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.239915 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.239889 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.240214 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.240194 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.240336 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.240304 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.240394 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.240377 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.240455 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.240441 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.260543 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.260522 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdgp\" (UniqueName: \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-kube-api-access-4jdgp\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.339305 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.339286 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.339375 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.339315 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.339462 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.339401 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.339462 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.339426 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.339564 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.339458 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.339564 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.339501 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.339564 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.339532 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.339564 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.339560 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj74n\" (UniqueName: \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-kube-api-access-cj74n\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.339811 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.339586 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.339914 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.339890 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.340058 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.340041 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.340144 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.340090 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.340144 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.340108 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.340262 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.340190 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7492ab70-7728-4bdf-ae23-9842106016fd-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.341149 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.341118 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.342392 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.342364 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.342648 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.342621 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.342756 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.342673 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.342830 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.342783 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.342891 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.342867 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7492ab70-7728-4bdf-ae23-9842106016fd-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.342891 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.342876 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.343055 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.343037 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.343573 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.343547 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.362467 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.362444 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:21.365837 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.365813 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj74n\" (UniqueName: \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-kube-api-access-cj74n\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:21.690640 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.690555 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 19:00:21.693979 ip-10-0-133-62 kubenswrapper[5014]: W1030 19:00:21.693951 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e91d58_487f_44fd_a972_4c872191374d.slice/crio-893e78b3319f8980ec6595d8c45133606e5e6d95a0e716c14a5b654c7b107c26 WatchSource:0}: Error finding container 893e78b3319f8980ec6595d8c45133606e5e6d95a0e716c14a5b654c7b107c26: Status 404 returned error can't find the container with id 893e78b3319f8980ec6595d8c45133606e5e6d95a0e716c14a5b654c7b107c26 Oct 30 19:00:21.961585 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:21.961523 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b4e91d58-487f-44fd-a972-4c872191374d","Type":"ContainerStarted","Data":"893e78b3319f8980ec6595d8c45133606e5e6d95a0e716c14a5b654c7b107c26"} Oct 30 19:00:22.969159 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:22.969114 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b4e91d58-487f-44fd-a972-4c872191374d","Type":"ContainerStarted","Data":"08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a"} Oct 30 19:00:25.987107 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:25.987070 5014 generic.go:356] "Generic (PLEG): container finished" podID="b4e91d58-487f-44fd-a972-4c872191374d" containerID="08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a" exitCode=0 Oct 30 19:00:25.987496 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:25.987148 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b4e91d58-487f-44fd-a972-4c872191374d","Type":"ContainerDied","Data":"08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a"} Oct 30 19:00:26.994635 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:26.994585 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b4e91d58-487f-44fd-a972-4c872191374d","Type":"ContainerStarted","Data":"7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113"} Oct 30 19:00:26.995007 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:26.994664 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b4e91d58-487f-44fd-a972-4c872191374d","Type":"ContainerStarted","Data":"bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1"} Oct 30 19:00:26.995007 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:26.994844 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:27.065844 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:27.065794 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.77892821 podStartE2EDuration="6.06578257s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.28685436s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:27.052972626 +0000 UTC m=+3840.041313112" watchObservedRunningTime="2026-10-30 19:00:27.06578257 +0000 UTC m=+3840.054123057" Oct 30 19:00:28.597582 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:28.597545 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:28.625878 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:28.625852 5014 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:28.698428 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:28.698402 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:28.698571 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:28.698549 5014 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:28.712622 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:28.712582 5014 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:28.815509 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:28.815485 5014 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/26c702426f1aeab2e9e6116e994cbd362b1230f690bb6d32002e36ee94fa3aec/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:28.842354 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:28.842334 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:28.944275 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:28.944223 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-72b7s\"" Oct 30 19:00:28.951334 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:28.951317 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:29.282705 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:29.282632 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 19:00:29.285920 ip-10-0-133-62 kubenswrapper[5014]: W1030 19:00:29.285888 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7492ab70_7728_4bdf_ae23_9842106016fd.slice/crio-2302f7eb22a5b3f7ceebd77914e0d742288e65314e262fa321fc23d7f49ad698 WatchSource:0}: Error finding container 2302f7eb22a5b3f7ceebd77914e0d742288e65314e262fa321fc23d7f49ad698: Status 404 returned error can't find the container with id 2302f7eb22a5b3f7ceebd77914e0d742288e65314e262fa321fc23d7f49ad698 Oct 30 19:00:30.011998 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:30.011968 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7492ab70-7728-4bdf-ae23-9842106016fd","Type":"ContainerStarted","Data":"767360bc3d32840440c72d7d1f3006ae9cac31f9cd889b1aaf858a6f9438478e"} Oct 30 19:00:30.012318 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:30.012044 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7492ab70-7728-4bdf-ae23-9842106016fd","Type":"ContainerStarted","Data":"2302f7eb22a5b3f7ceebd77914e0d742288e65314e262fa321fc23d7f49ad698"} Oct 30 19:00:33.000797 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:33.000766 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:34.032248 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:34.032214 5014 generic.go:356] "Generic (PLEG): container finished" podID="7492ab70-7728-4bdf-ae23-9842106016fd" containerID="767360bc3d32840440c72d7d1f3006ae9cac31f9cd889b1aaf858a6f9438478e" exitCode=0 Oct 30 19:00:34.032682 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:34.032254 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7492ab70-7728-4bdf-ae23-9842106016fd","Type":"ContainerDied","Data":"767360bc3d32840440c72d7d1f3006ae9cac31f9cd889b1aaf858a6f9438478e"} Oct 30 19:00:35.039423 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:35.039388 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7492ab70-7728-4bdf-ae23-9842106016fd","Type":"ContainerStarted","Data":"b6e8bbdcad4696466eb1df1debee8b413480433948d55fea643527be774021e2"} Oct 30 19:00:35.039850 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:35.039452 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7492ab70-7728-4bdf-ae23-9842106016fd","Type":"ContainerStarted","Data":"53e64235c36a76bab286c61dcd18682e8e3ec968f149e353d6524a750218294c"} Oct 30 19:00:35.039850 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:35.039474 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7492ab70-7728-4bdf-ae23-9842106016fd","Type":"ContainerStarted","Data":"c14edb5063ea4c102376ec705969b96c70630260156903f01595cc88b6429f45"} Oct 30 19:00:35.118066 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:35.118014 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=10.634090033 podStartE2EDuration="14.118001872s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.483911839s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:35.104884607 +0000 UTC m=+3848.093225094" watchObservedRunningTime="2026-10-30 19:00:35.118001872 +0000 UTC m=+3848.106342358" Oct 30 19:00:38.952191 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:38.952161 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:43.333361 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:00:43.333323 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 19:00:43.952470 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:43.952435 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:43.958055 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:43.958034 5014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:44.085988 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:44.085966 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:50.098577 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.098522 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl"] Oct 30 19:00:50.099125 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.098961 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" podUID="0b7e0f15-c382-45ba-9684-5b0074548309" containerName="kube-rbac-proxy" containerID="cri-o://fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd" gracePeriod=30 Oct 30 19:00:50.110950 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.110894 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5"] Oct 30 19:00:50.111317 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.111257 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="kube-rbac-proxy" containerID="cri-o://53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b" gracePeriod=30 Oct 30 19:00:50.111476 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.111409 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="method-gate" containerID="cri-o://d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe" gracePeriod=30 Oct 30 19:00:50.111476 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.111422 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="prom-label-proxy" containerID="cri-o://7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c" gracePeriod=30 Oct 30 19:00:50.390294 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.390266 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:50.393787 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.393767 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:50.461580 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.461530 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-prometheus-tls-ca\") pod \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " Oct 30 19:00:50.461756 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.461583 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7e0f15-c382-45ba-9684-5b0074548309-kube-api-access-x9z9h\" (UniqueName: \"kubernetes.io/projected/0b7e0f15-c382-45ba-9684-5b0074548309-kube-api-access-x9z9h\") pod \"0b7e0f15-c382-45ba-9684-5b0074548309\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " Oct 30 19:00:50.461756 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.461649 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-config\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-config\") pod \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " Oct 30 19:00:50.461756 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.461683 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-kube-rbac-proxy-config\") pod \"0b7e0f15-c382-45ba-9684-5b0074548309\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " Oct 30 19:00:50.461944 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.461852 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-tls\" (UniqueName: \"kubernetes.io/secret/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-tls\") pod \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " Oct 30 19:00:50.461944 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.461907 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-client-cert\") pod \"0b7e0f15-c382-45ba-9684-5b0074548309\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " Oct 30 19:00:50.462062 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.461951 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-client-ca\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-client-ca\") pod \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " Oct 30 19:00:50.462062 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.461959 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-prometheus-tls-ca" pod "102f1fe2-9102-426c-9c6c-1c63c7e7dad8" (UID: "102f1fe2-9102-426c-9c6c-1c63c7e7dad8"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:50.462062 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.461991 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-kube-api-access-d7f5f\" (UniqueName: \"kubernetes.io/projected/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-kube-api-access-d7f5f\") pod \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\" (UID: \"102f1fe2-9102-426c-9c6c-1c63c7e7dad8\") " Oct 30 19:00:50.462062 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.462039 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-tls\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-tls\") pod \"0b7e0f15-c382-45ba-9684-5b0074548309\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " Oct 30 19:00:50.462295 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.462080 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-ca\") pod \"0b7e0f15-c382-45ba-9684-5b0074548309\" (UID: \"0b7e0f15-c382-45ba-9684-5b0074548309\") " Oct 30 19:00:50.463448 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.462366 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-prometheus-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.463448 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.462376 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-client-ca" pod "102f1fe2-9102-426c-9c6c-1c63c7e7dad8" (UID: "102f1fe2-9102-426c-9c6c-1c63c7e7dad8"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:50.463448 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.462395 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-config" pod "102f1fe2-9102-426c-9c6c-1c63c7e7dad8" (UID: "102f1fe2-9102-426c-9c6c-1c63c7e7dad8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:50.464516 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.464495 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-ca" pod "0b7e0f15-c382-45ba-9684-5b0074548309" (UID: "0b7e0f15-c382-45ba-9684-5b0074548309"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:50.465632 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.465467 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-client-cert" pod "0b7e0f15-c382-45ba-9684-5b0074548309" (UID: "0b7e0f15-c382-45ba-9684-5b0074548309"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:50.465632 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.465542 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-tls" pod "102f1fe2-9102-426c-9c6c-1c63c7e7dad8" (UID: "102f1fe2-9102-426c-9c6c-1c63c7e7dad8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:50.465632 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.465604 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-kube-rbac-proxy-config" pod "0b7e0f15-c382-45ba-9684-5b0074548309" (UID: "0b7e0f15-c382-45ba-9684-5b0074548309"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:50.466126 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.466101 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7e0f15-c382-45ba-9684-5b0074548309-kube-api-access-x9z9h" pod "0b7e0f15-c382-45ba-9684-5b0074548309" (UID: "0b7e0f15-c382-45ba-9684-5b0074548309"). InnerVolumeSpecName "kube-api-access-x9z9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:50.466392 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.466359 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-kube-api-access-d7f5f" pod "102f1fe2-9102-426c-9c6c-1c63c7e7dad8" (UID: "102f1fe2-9102-426c-9c6c-1c63c7e7dad8"). InnerVolumeSpecName "kube-api-access-d7f5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:50.466514 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.466487 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-tls" pod "0b7e0f15-c382-45ba-9684-5b0074548309" (UID: "0b7e0f15-c382-45ba-9684-5b0074548309"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:50.471208 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.470714 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst"] Oct 30 19:00:50.471208 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.470918 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" podUID="4845506f-722e-4a85-abbb-f3bb3801d001" containerName="thanos-querier" containerID="cri-o://dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c" gracePeriod=30 Oct 30 19:00:50.564189 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.564156 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9z9h\" (UniqueName: \"kubernetes.io/projected/0b7e0f15-c382-45ba-9684-5b0074548309-kube-api-access-x9z9h\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.564189 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.564187 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.564409 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.564201 5014 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-kube-rbac-proxy-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.564409 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.564212 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.564409 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.564223 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.564409 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.564234 5014 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-client-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.564409 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.564245 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7f5f\" (UniqueName: \"kubernetes.io/projected/102f1fe2-9102-426c-9c6c-1c63c7e7dad8-kube-api-access-d7f5f\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.564409 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.564256 5014 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0b7e0f15-c382-45ba-9684-5b0074548309-tls\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.564409 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.564267 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/0b7e0f15-c382-45ba-9684-5b0074548309-prometheus-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.604571 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.604533 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 19:00:50.605172 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.604866 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="prometheus" containerID="cri-o://c14edb5063ea4c102376ec705969b96c70630260156903f01595cc88b6429f45" gracePeriod=600 Oct 30 19:00:50.605172 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.604911 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="config-reloader" containerID="cri-o://53e64235c36a76bab286c61dcd18682e8e3ec968f149e353d6524a750218294c" gracePeriod=600 Oct 30 19:00:50.605363 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.604960 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="thanos-sidecar" containerID="cri-o://b6e8bbdcad4696466eb1df1debee8b413480433948d55fea643527be774021e2" gracePeriod=600 Oct 30 19:00:50.622016 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.616283 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 19:00:50.622016 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.616576 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="alertmanager" containerID="cri-o://bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1" gracePeriod=120 Oct 30 19:00:50.622016 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.616678 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="config-reloader" containerID="cri-o://7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113" gracePeriod=120 Oct 30 19:00:50.735659 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.735638 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" Oct 30 19:00:50.765806 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.765779 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4845506f-722e-4a85-abbb-f3bb3801d001-kube-api-access-4455l\" (UniqueName: \"kubernetes.io/projected/4845506f-722e-4a85-abbb-f3bb3801d001-kube-api-access-4455l\") pod \"4845506f-722e-4a85-abbb-f3bb3801d001\" (UID: \"4845506f-722e-4a85-abbb-f3bb3801d001\") " Oct 30 19:00:50.767714 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.767692 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4845506f-722e-4a85-abbb-f3bb3801d001-kube-api-access-4455l" pod "4845506f-722e-4a85-abbb-f3bb3801d001" (UID: "4845506f-722e-4a85-abbb-f3bb3801d001"). InnerVolumeSpecName "kube-api-access-4455l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:50.866562 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:50.866534 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4455l\" (UniqueName: \"kubernetes.io/projected/4845506f-722e-4a85-abbb-f3bb3801d001-kube-api-access-4455l\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.121383 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.121358 5014 generic.go:356] "Generic (PLEG): container finished" podID="b4e91d58-487f-44fd-a972-4c872191374d" containerID="7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113" exitCode=0 Oct 30 19:00:51.121740 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.121435 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b4e91d58-487f-44fd-a972-4c872191374d","Type":"ContainerDied","Data":"7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113"} Oct 30 19:00:51.122657 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.122637 5014 generic.go:356] "Generic (PLEG): container finished" podID="0b7e0f15-c382-45ba-9684-5b0074548309" containerID="fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd" exitCode=0 Oct 30 19:00:51.122769 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.122692 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" Oct 30 19:00:51.122769 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.122721 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" event={"ID":"0b7e0f15-c382-45ba-9684-5b0074548309","Type":"ContainerDied","Data":"fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd"} Oct 30 19:00:51.122769 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.122764 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl" event={"ID":"0b7e0f15-c382-45ba-9684-5b0074548309","Type":"ContainerDied","Data":"e3b68516c6bcea75607531ff8f4e5b5f20a5477869a0f22dcdbfe8bf04f8a74c"} Oct 30 19:00:51.122910 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.122795 5014 scope.go:122] "RemoveContainer" containerID="fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd" Oct 30 19:00:51.125784 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.125762 5014 generic.go:356] "Generic (PLEG): container finished" podID="7492ab70-7728-4bdf-ae23-9842106016fd" containerID="b6e8bbdcad4696466eb1df1debee8b413480433948d55fea643527be774021e2" exitCode=0 Oct 30 19:00:51.125867 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.125784 5014 generic.go:356] "Generic (PLEG): container finished" podID="7492ab70-7728-4bdf-ae23-9842106016fd" containerID="53e64235c36a76bab286c61dcd18682e8e3ec968f149e353d6524a750218294c" exitCode=0 Oct 30 19:00:51.125867 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.125797 5014 generic.go:356] "Generic (PLEG): container finished" podID="7492ab70-7728-4bdf-ae23-9842106016fd" containerID="c14edb5063ea4c102376ec705969b96c70630260156903f01595cc88b6429f45" exitCode=0 Oct 30 19:00:51.125867 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.125845 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7492ab70-7728-4bdf-ae23-9842106016fd","Type":"ContainerDied","Data":"b6e8bbdcad4696466eb1df1debee8b413480433948d55fea643527be774021e2"} Oct 30 19:00:51.126054 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.125916 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7492ab70-7728-4bdf-ae23-9842106016fd","Type":"ContainerDied","Data":"53e64235c36a76bab286c61dcd18682e8e3ec968f149e353d6524a750218294c"} Oct 30 19:00:51.126054 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.125956 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7492ab70-7728-4bdf-ae23-9842106016fd","Type":"ContainerDied","Data":"c14edb5063ea4c102376ec705969b96c70630260156903f01595cc88b6429f45"} Oct 30 19:00:51.127116 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.127096 5014 generic.go:356] "Generic (PLEG): container finished" podID="4845506f-722e-4a85-abbb-f3bb3801d001" containerID="dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c" exitCode=0 Oct 30 19:00:51.127208 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.127135 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" event={"ID":"4845506f-722e-4a85-abbb-f3bb3801d001","Type":"ContainerDied","Data":"dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c"} Oct 30 19:00:51.127208 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.127151 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" Oct 30 19:00:51.127208 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.127171 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst" event={"ID":"4845506f-722e-4a85-abbb-f3bb3801d001","Type":"ContainerDied","Data":"526fad875e589ed1734e8a22f3d1462fe8e1008208cd6af56285df8542592dc7"} Oct 30 19:00:51.129180 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.129162 5014 generic.go:356] "Generic (PLEG): container finished" podID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerID="7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c" exitCode=0 Oct 30 19:00:51.129180 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.129180 5014 generic.go:356] "Generic (PLEG): container finished" podID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerID="d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe" exitCode=0 Oct 30 19:00:51.129311 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.129187 5014 generic.go:356] "Generic (PLEG): container finished" podID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerID="53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b" exitCode=0 Oct 30 19:00:51.129311 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.129206 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" event={"ID":"102f1fe2-9102-426c-9c6c-1c63c7e7dad8","Type":"ContainerDied","Data":"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c"} Oct 30 19:00:51.129311 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.129229 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" Oct 30 19:00:51.129311 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.129241 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" event={"ID":"102f1fe2-9102-426c-9c6c-1c63c7e7dad8","Type":"ContainerDied","Data":"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe"} Oct 30 19:00:51.129311 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.129291 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" event={"ID":"102f1fe2-9102-426c-9c6c-1c63c7e7dad8","Type":"ContainerDied","Data":"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b"} Oct 30 19:00:51.129487 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.129316 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5" event={"ID":"102f1fe2-9102-426c-9c6c-1c63c7e7dad8","Type":"ContainerDied","Data":"5a419ce8e30740649bee5d0461562ac7cf8305866ceab6a3c3f6c8d54731b34d"} Oct 30 19:00:51.133661 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.133645 5014 scope.go:122] "RemoveContainer" containerID="fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd" Oct 30 19:00:51.133938 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:00:51.133921 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd\": container with ID starting with fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd not found: ID does not exist" containerID="fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd" Oct 30 19:00:51.133989 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.133946 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd"} err="failed to get container status \"fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd\": rpc error: code = NotFound desc = could not find container \"fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd\": container with ID starting with fea340449746ab09068e2bcdef93652f28af181a2a2b7db8aeb90b16761548cd not found: ID does not exist" Oct 30 19:00:51.133989 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.133962 5014 scope.go:122] "RemoveContainer" containerID="dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c" Oct 30 19:00:51.143254 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.143233 5014 scope.go:122] "RemoveContainer" containerID="dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c" Oct 30 19:00:51.143498 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:00:51.143482 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c\": container with ID starting with dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c not found: ID does not exist" containerID="dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c" Oct 30 19:00:51.143573 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.143503 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c"} err="failed to get container status \"dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c\": rpc error: code = NotFound desc = could not find container \"dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c\": container with ID starting with dadd4a099d92060e97a004b5c4ee0ad7a3a36a747c4b54a82eca2bb1c246b92c not found: ID does not exist" Oct 30 19:00:51.143573 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.143518 5014 scope.go:122] "RemoveContainer" containerID="7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c" Oct 30 19:00:51.151488 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.151471 5014 scope.go:122] "RemoveContainer" containerID="d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe" Oct 30 19:00:51.158903 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.158885 5014 scope.go:122] "RemoveContainer" containerID="53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b" Oct 30 19:00:51.166230 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.166215 5014 scope.go:122] "RemoveContainer" containerID="7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c" Oct 30 19:00:51.166481 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:00:51.166465 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c\": container with ID starting with 7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c not found: ID does not exist" containerID="7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c" Oct 30 19:00:51.166542 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.166487 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c"} err="failed to get container status \"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c\": rpc error: code = NotFound desc = could not find container \"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c\": container with ID starting with 7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c not found: ID does not exist" Oct 30 19:00:51.166542 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.166503 5014 scope.go:122] "RemoveContainer" containerID="d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe" Oct 30 19:00:51.166750 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:00:51.166734 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe\": container with ID starting with d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe not found: ID does not exist" containerID="d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe" Oct 30 19:00:51.166801 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.166753 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe"} err="failed to get container status \"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe\": rpc error: code = NotFound desc = could not find container \"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe\": container with ID starting with d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe not found: ID does not exist" Oct 30 19:00:51.166801 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.166765 5014 scope.go:122] "RemoveContainer" containerID="53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b" Oct 30 19:00:51.166990 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:00:51.166972 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b\": container with ID starting with 53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b not found: ID does not exist" containerID="53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b" Oct 30 19:00:51.167070 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.166999 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b"} err="failed to get container status \"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b\": rpc error: code = NotFound desc = could not find container \"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b\": container with ID starting with 53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b not found: ID does not exist" Oct 30 19:00:51.167070 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.167021 5014 scope.go:122] "RemoveContainer" containerID="7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c" Oct 30 19:00:51.167234 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.167212 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c"} err="failed to get container status \"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c\": rpc error: code = NotFound desc = could not find container \"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c\": container with ID starting with 7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c not found: ID does not exist" Oct 30 19:00:51.167279 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.167235 5014 scope.go:122] "RemoveContainer" containerID="d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe" Oct 30 19:00:51.167456 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.167439 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe"} err="failed to get container status \"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe\": rpc error: code = NotFound desc = could not find container \"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe\": container with ID starting with d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe not found: ID does not exist" Oct 30 19:00:51.167517 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.167457 5014 scope.go:122] "RemoveContainer" containerID="53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b" Oct 30 19:00:51.167681 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.167665 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b"} err="failed to get container status \"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b\": rpc error: code = NotFound desc = could not find container \"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b\": container with ID starting with 53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b not found: ID does not exist" Oct 30 19:00:51.167751 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.167681 5014 scope.go:122] "RemoveContainer" containerID="7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c" Oct 30 19:00:51.167966 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.167942 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c"} err="failed to get container status \"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c\": rpc error: code = NotFound desc = could not find container \"7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c\": container with ID starting with 7772e4aec48ca557b07b4b20e4859f3ae729df4521583a134cd8ad7b64788a8c not found: ID does not exist" Oct 30 19:00:51.167966 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.167966 5014 scope.go:122] "RemoveContainer" containerID="d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe" Oct 30 19:00:51.168208 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.168191 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe"} err="failed to get container status \"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe\": rpc error: code = NotFound desc = could not find container \"d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe\": container with ID starting with d54c48291db0b2a85fd013e1cf2dc99537de94ecfb4566e41d96463f3f65defe not found: ID does not exist" Oct 30 19:00:51.168265 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.168208 5014 scope.go:122] "RemoveContainer" containerID="53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b" Oct 30 19:00:51.168424 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.168407 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b"} err="failed to get container status \"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b\": rpc error: code = NotFound desc = could not find container \"53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b\": container with ID starting with 53752c0581fd64d152f94cf1893fae906ae85fd3f292d1fd3964121f262cbe1b not found: ID does not exist" Oct 30 19:00:51.173284 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.173264 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl"] Oct 30 19:00:51.180018 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.179999 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2vkpl"] Oct 30 19:00:51.241928 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.241907 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst"] Oct 30 19:00:51.244567 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.244543 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-hrvst"] Oct 30 19:00:51.281072 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.281043 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5"] Oct 30 19:00:51.289945 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.288572 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-tfmn5"] Oct 30 19:00:51.300170 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.300136 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 19:00:51.300473 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.300321 5014 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="99afabe9-bc71-4663-a085-70939405f3dd" containerName="otc-container" containerID="cri-o://ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c" gracePeriod=30 Oct 30 19:00:51.434451 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.434429 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:51.472045 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472016 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-secret-prometheus-web-tls-ca\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472229 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472079 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472229 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472143 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-tls-assets\" (UniqueName: \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-tls-assets\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472229 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472188 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-config\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-config\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472420 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472229 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472420 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472264 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472420 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472296 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472420 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472403 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472688 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472458 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-thanos-prometheus-http-client-file\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472688 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472498 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472688 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472536 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7492ab70-7728-4bdf-ae23-9842106016fd-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7492ab70-7728-4bdf-ae23-9842106016fd-config-out\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472688 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472587 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-kube-api-access-cj74n\" (UniqueName: \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-kube-api-access-cj74n\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472688 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472664 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"7492ab70-7728-4bdf-ae23-9842106016fd\" (UID: \"7492ab70-7728-4bdf-ae23-9842106016fd\") " Oct 30 19:00:51.472962 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.472818 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-1" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:51.473162 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.473136 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-0" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:51.473504 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.473473 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.473624 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.473502 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.474698 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.474662 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-2" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:51.475750 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.475676 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.475953 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.475917 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-tls-assets" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:51.476038 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.475921 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-secret-prometheus-web-tls-ca" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.476691 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.476650 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.477417 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.477381 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-config" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.477944 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.477914 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-thanos-prometheus-http-client-file" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.478059 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.478041 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7492ab70-7728-4bdf-ae23-9842106016fd-config-out" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:51.478257 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.478234 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-kube-api-access-cj74n" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "kube-api-access-cj74n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:51.479185 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.479164 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "pvc-c02e8472-bb1a-4974-aa89-86eef715889b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 19:00:51.493834 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.493797 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config" pod "7492ab70-7728-4bdf-ae23-9842106016fd" (UID: "7492ab70-7728-4bdf-ae23-9842106016fd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.573148 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.573120 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:51.574929 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.574902 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj74n\" (UniqueName: \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-kube-api-access-cj74n\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.575023 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.574935 5014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.575023 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.574954 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.575023 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.574971 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7492ab70-7728-4bdf-ae23-9842106016fd-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.575023 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.574987 5014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.575023 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.574999 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.575023 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.575013 5014 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.575238 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.575050 5014 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" " Oct 30 19:00:51.575238 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.575068 5014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7492ab70-7728-4bdf-ae23-9842106016fd-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.575238 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.575083 5014 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7492ab70-7728-4bdf-ae23-9842106016fd-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.575238 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.575097 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7492ab70-7728-4bdf-ae23-9842106016fd-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.587184 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.587147 5014 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c02e8472-bb1a-4974-aa89-86eef715889b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e") on node "ip-10-0-133-62.ec2.internal" Oct 30 19:00:51.675943 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.675915 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-tls-certs\") pod \"99afabe9-bc71-4663-a085-70939405f3dd\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " Oct 30 19:00:51.676111 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.675956 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-data-science-collector-ta-client-cert\") pod \"99afabe9-bc71-4663-a085-70939405f3dd\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " Oct 30 19:00:51.676111 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.675989 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99afabe9-bc71-4663-a085-70939405f3dd-kube-api-access-55fdx\" (UniqueName: \"kubernetes.io/projected/99afabe9-bc71-4663-a085-70939405f3dd-kube-api-access-55fdx\") pod \"99afabe9-bc71-4663-a085-70939405f3dd\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " Oct 30 19:00:51.676111 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.676046 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99afabe9-bc71-4663-a085-70939405f3dd-otc-internal\" (UniqueName: \"kubernetes.io/configmap/99afabe9-bc71-4663-a085-70939405f3dd-otc-internal\") pod \"99afabe9-bc71-4663-a085-70939405f3dd\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " Oct 30 19:00:51.676111 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.676100 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-otlp-tls-certs\") pod \"99afabe9-bc71-4663-a085-70939405f3dd\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " Oct 30 19:00:51.676365 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.676146 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-monitoring-tls-certs\") pod \"99afabe9-bc71-4663-a085-70939405f3dd\" (UID: \"99afabe9-bc71-4663-a085-70939405f3dd\") " Oct 30 19:00:51.676473 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.676425 5014 reconciler_common.go:299] "Volume detached for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"csi-64cec75a00cb543cfbf0c769dcc1dfc6f9771ef0ed63d1e267150bd32533fbbf\"" Oct 30 19:00:51.676908 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.676882 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99afabe9-bc71-4663-a085-70939405f3dd-otc-internal" pod "99afabe9-bc71-4663-a085-70939405f3dd" (UID: "99afabe9-bc71-4663-a085-70939405f3dd"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:51.678435 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.678391 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-data-science-collector-ta-client-cert" pod "99afabe9-bc71-4663-a085-70939405f3dd" (UID: "99afabe9-bc71-4663-a085-70939405f3dd"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.678921 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.678897 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-monitoring-tls-certs" pod "99afabe9-bc71-4663-a085-70939405f3dd" (UID: "99afabe9-bc71-4663-a085-70939405f3dd"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.679019 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.678922 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-otlp-tls-certs" pod "99afabe9-bc71-4663-a085-70939405f3dd" (UID: "99afabe9-bc71-4663-a085-70939405f3dd"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.679019 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.678900 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99afabe9-bc71-4663-a085-70939405f3dd-kube-api-access-55fdx" pod "99afabe9-bc71-4663-a085-70939405f3dd" (UID: "99afabe9-bc71-4663-a085-70939405f3dd"). InnerVolumeSpecName "kube-api-access-55fdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:51.679019 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.678996 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-tls-certs" pod "99afabe9-bc71-4663-a085-70939405f3dd" (UID: "99afabe9-bc71-4663-a085-70939405f3dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.777842 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.777764 5014 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/99afabe9-bc71-4663-a085-70939405f3dd-otc-internal\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.777842 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.777797 5014 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-otlp-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.777842 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.777808 5014 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-monitoring-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.777842 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.777817 5014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-tls-certs\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.777842 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.777826 5014 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/99afabe9-bc71-4663-a085-70939405f3dd-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.777842 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:51.777835 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55fdx\" (UniqueName: \"kubernetes.io/projected/99afabe9-bc71-4663-a085-70939405f3dd-kube-api-access-55fdx\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:00:52.140785 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.140744 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7492ab70-7728-4bdf-ae23-9842106016fd","Type":"ContainerDied","Data":"2302f7eb22a5b3f7ceebd77914e0d742288e65314e262fa321fc23d7f49ad698"} Oct 30 19:00:52.141387 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.140829 5014 scope.go:122] "RemoveContainer" containerID="b6e8bbdcad4696466eb1df1debee8b413480433948d55fea643527be774021e2" Oct 30 19:00:52.141387 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.140756 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 19:00:52.143205 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.143158 5014 generic.go:356] "Generic (PLEG): container finished" podID="99afabe9-bc71-4663-a085-70939405f3dd" containerID="ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c" exitCode=0 Oct 30 19:00:52.143205 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.143188 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"99afabe9-bc71-4663-a085-70939405f3dd","Type":"ContainerDied","Data":"ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c"} Oct 30 19:00:52.143205 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.143206 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 19:00:52.143436 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.143235 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"99afabe9-bc71-4663-a085-70939405f3dd","Type":"ContainerDied","Data":"4fa04489f3e785ca9beea95d98d1bfecf867aef0a43f27ce11552b5a3d62b4d6"} Oct 30 19:00:52.151228 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.151206 5014 scope.go:122] "RemoveContainer" containerID="53e64235c36a76bab286c61dcd18682e8e3ec968f149e353d6524a750218294c" Oct 30 19:00:52.161354 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.161335 5014 scope.go:122] "RemoveContainer" containerID="c14edb5063ea4c102376ec705969b96c70630260156903f01595cc88b6429f45" Oct 30 19:00:52.169625 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.169587 5014 scope.go:122] "RemoveContainer" containerID="767360bc3d32840440c72d7d1f3006ae9cac31f9cd889b1aaf858a6f9438478e" Oct 30 19:00:52.179558 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.179539 5014 scope.go:122] "RemoveContainer" containerID="ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c" Oct 30 19:00:52.196106 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.196083 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 19:00:52.205623 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.202791 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 19:00:52.214265 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.214242 5014 scope.go:122] "RemoveContainer" containerID="ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c" Oct 30 19:00:52.214695 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:00:52.214660 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c\": container with ID starting with ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c not found: ID does not exist" containerID="ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c" Oct 30 19:00:52.214784 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.214702 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c"} err="failed to get container status \"ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c\": rpc error: code = NotFound desc = could not find container \"ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c\": container with ID starting with ed29c1cb02413fee3393af82efe05d1ccaf8fc544b04bbb6b59fffc451a48b8c not found: ID does not exist" Oct 30 19:00:52.238629 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.238581 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 19:00:52.246576 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.246554 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 19:00:52.999916 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:52.999876 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.177:9093/-/ready\": dial tcp 10.132.0.177:9093: connect: connection refused" Oct 30 19:00:53.148471 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:53.148438 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7e0f15-c382-45ba-9684-5b0074548309" path="/var/lib/kubelet/pods/0b7e0f15-c382-45ba-9684-5b0074548309/volumes" Oct 30 19:00:53.148967 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:53.148947 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" path="/var/lib/kubelet/pods/102f1fe2-9102-426c-9c6c-1c63c7e7dad8/volumes" Oct 30 19:00:53.149449 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:53.149431 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4845506f-722e-4a85-abbb-f3bb3801d001" path="/var/lib/kubelet/pods/4845506f-722e-4a85-abbb-f3bb3801d001/volumes" Oct 30 19:00:53.150304 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:53.150285 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" path="/var/lib/kubelet/pods/7492ab70-7728-4bdf-ae23-9842106016fd/volumes" Oct 30 19:00:53.150823 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:53.150808 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99afabe9-bc71-4663-a085-70939405f3dd" path="/var/lib/kubelet/pods/99afabe9-bc71-4663-a085-70939405f3dd/volumes" Oct 30 19:00:58.000732 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:00:58.000692 5014 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.177:9093/-/ready\": dial tcp 10.132.0.177:9093: connect: connection refused" Oct 30 19:01:00.867606 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.867575 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:01:00.951391 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.951324 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-web-config\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-web-config\") pod \"b4e91d58-487f-44fd-a972-4c872191374d\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " Oct 30 19:01:00.951391 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.951379 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-config-out\") pod \"b4e91d58-487f-44fd-a972-4c872191374d\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " Oct 30 19:01:00.951647 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.951420 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-config-volume\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-config-volume\") pod \"b4e91d58-487f-44fd-a972-4c872191374d\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " Oct 30 19:01:00.951647 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.951464 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-kube-api-access-4jdgp\" (UniqueName: \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-kube-api-access-4jdgp\") pod \"b4e91d58-487f-44fd-a972-4c872191374d\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " Oct 30 19:01:00.951647 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.951498 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-cluster-tls-config\") pod \"b4e91d58-487f-44fd-a972-4c872191374d\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " Oct 30 19:01:00.951647 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.951524 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-tls-assets\" (UniqueName: \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-tls-assets\") pod \"b4e91d58-487f-44fd-a972-4c872191374d\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " Oct 30 19:01:00.951647 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.951561 5014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-alertmanager-data-science-monitoringstack-db\") pod \"b4e91d58-487f-44fd-a972-4c872191374d\" (UID: \"b4e91d58-487f-44fd-a972-4c872191374d\") " Oct 30 19:01:00.952131 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.952107 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-alertmanager-data-science-monitoringstack-db" pod "b4e91d58-487f-44fd-a972-4c872191374d" (UID: "b4e91d58-487f-44fd-a972-4c872191374d"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:00.954258 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.954220 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-config-volume" pod "b4e91d58-487f-44fd-a972-4c872191374d" (UID: "b4e91d58-487f-44fd-a972-4c872191374d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:00.954258 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.954234 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-kube-api-access-4jdgp" pod "b4e91d58-487f-44fd-a972-4c872191374d" (UID: "b4e91d58-487f-44fd-a972-4c872191374d"). InnerVolumeSpecName "kube-api-access-4jdgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:00.954419 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.954336 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-config-out" pod "b4e91d58-487f-44fd-a972-4c872191374d" (UID: "b4e91d58-487f-44fd-a972-4c872191374d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:00.954543 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.954519 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-tls-assets" pod "b4e91d58-487f-44fd-a972-4c872191374d" (UID: "b4e91d58-487f-44fd-a972-4c872191374d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:00.959537 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.959414 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-cluster-tls-config" pod "b4e91d58-487f-44fd-a972-4c872191374d" (UID: "b4e91d58-487f-44fd-a972-4c872191374d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:00.965977 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:00.965954 5014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-web-config" pod "b4e91d58-487f-44fd-a972-4c872191374d" (UID: "b4e91d58-487f-44fd-a972-4c872191374d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:01.052350 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.052311 5014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-web-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:01:01.052350 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.052351 5014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-config-out\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:01:01.052350 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.052361 5014 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-config-volume\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:01:01.052515 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.052369 5014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdgp\" (UniqueName: \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-kube-api-access-4jdgp\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:01:01.052515 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.052378 5014 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4e91d58-487f-44fd-a972-4c872191374d-cluster-tls-config\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:01:01.052515 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.052387 5014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4e91d58-487f-44fd-a972-4c872191374d-tls-assets\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:01:01.052515 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.052396 5014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d58-487f-44fd-a972-4c872191374d-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-62.ec2.internal\" DevicePath \"\"" Oct 30 19:01:01.186589 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.186561 5014 generic.go:356] "Generic (PLEG): container finished" podID="b4e91d58-487f-44fd-a972-4c872191374d" containerID="bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1" exitCode=0 Oct 30 19:01:01.186759 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.186632 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b4e91d58-487f-44fd-a972-4c872191374d","Type":"ContainerDied","Data":"bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1"} Oct 30 19:01:01.186759 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.186653 5014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:01:01.186759 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.186707 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"b4e91d58-487f-44fd-a972-4c872191374d","Type":"ContainerDied","Data":"893e78b3319f8980ec6595d8c45133606e5e6d95a0e716c14a5b654c7b107c26"} Oct 30 19:01:01.186902 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.186777 5014 scope.go:122] "RemoveContainer" containerID="7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113" Oct 30 19:01:01.195356 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.195331 5014 scope.go:122] "RemoveContainer" containerID="bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1" Oct 30 19:01:01.203225 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.203211 5014 scope.go:122] "RemoveContainer" containerID="08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a" Oct 30 19:01:01.210700 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.210684 5014 scope.go:122] "RemoveContainer" containerID="7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113" Oct 30 19:01:01.210993 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:01:01.210974 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113\": container with ID starting with 7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113 not found: ID does not exist" containerID="7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113" Oct 30 19:01:01.211080 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.211004 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113"} err="failed to get container status \"7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113\": rpc error: code = NotFound desc = could not find container \"7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113\": container with ID starting with 7fb394cae444c2708b04576a176e4f8191e364c460d69f1847cf9d4cf2d48113 not found: ID does not exist" Oct 30 19:01:01.211080 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.211026 5014 scope.go:122] "RemoveContainer" containerID="bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1" Oct 30 19:01:01.211299 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:01:01.211280 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1\": container with ID starting with bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1 not found: ID does not exist" containerID="bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1" Oct 30 19:01:01.211350 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.211305 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1"} err="failed to get container status \"bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1\": rpc error: code = NotFound desc = could not find container \"bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1\": container with ID starting with bdf536f924cbc788456a51f3751c50984d6e0a22c7e78cd8604b713c3112ffe1 not found: ID does not exist" Oct 30 19:01:01.211350 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.211321 5014 scope.go:122] "RemoveContainer" containerID="08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a" Oct 30 19:01:01.211524 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:01:01.211510 5014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a\": container with ID starting with 08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a not found: ID does not exist" containerID="08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a" Oct 30 19:01:01.211603 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.211529 5014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a"} err="failed to get container status \"08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a\": rpc error: code = NotFound desc = could not find container \"08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a\": container with ID starting with 08fbf23389614a9245e9ef52e9deec48e5bee31d8d484b20825016960386f97a not found: ID does not exist" Oct 30 19:01:01.238497 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.238475 5014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 19:01:01.250321 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:01.250304 5014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 19:01:03.142163 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:03.142111 5014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e91d58-487f-44fd-a972-4c872191374d" path="/var/lib/kubelet/pods/b4e91d58-487f-44fd-a972-4c872191374d/volumes" Oct 30 19:01:13.334003 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:01:13.333965 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 19:01:18.979482 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979444 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ht9cb/must-gather-jbttc"] Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979859 5014 cpu_manager.go:414] "containerMap: removing container" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="kube-rbac-proxy" Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979876 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="kube-rbac-proxy" Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979886 5014 cpu_manager.go:414] "containerMap: removing container" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="config-reloader" Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979892 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="config-reloader" Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979900 5014 cpu_manager.go:414] "containerMap: removing container" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="thanos-sidecar" Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979905 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="thanos-sidecar" Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979913 5014 cpu_manager.go:414] "containerMap: removing container" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="prom-label-proxy" Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979918 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="prom-label-proxy" Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979939 5014 cpu_manager.go:414] "containerMap: removing container" podUID="0b7e0f15-c382-45ba-9684-5b0074548309" containerName="kube-rbac-proxy" Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979944 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7e0f15-c382-45ba-9684-5b0074548309" containerName="kube-rbac-proxy" Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979955 5014 cpu_manager.go:414] "containerMap: removing container" podUID="4845506f-722e-4a85-abbb-f3bb3801d001" containerName="thanos-querier" Oct 30 19:01:18.979952 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979959 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4845506f-722e-4a85-abbb-f3bb3801d001" containerName="thanos-querier" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979966 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="config-reloader" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979970 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="config-reloader" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979979 5014 cpu_manager.go:414] "containerMap: removing container" podUID="99afabe9-bc71-4663-a085-70939405f3dd" containerName="otc-container" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979984 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99afabe9-bc71-4663-a085-70939405f3dd" containerName="otc-container" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979990 5014 cpu_manager.go:414] "containerMap: removing container" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="prometheus" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.979994 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="prometheus" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980005 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="alertmanager" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980013 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="alertmanager" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980026 5014 cpu_manager.go:414] "containerMap: removing container" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="init-config-reloader" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980031 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="init-config-reloader" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980038 5014 cpu_manager.go:414] "containerMap: removing container" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="init-config-reloader" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980043 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="init-config-reloader" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980054 5014 cpu_manager.go:414] "containerMap: removing container" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="method-gate" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980058 5014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="method-gate" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980118 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7e0f15-c382-45ba-9684-5b0074548309" containerName="kube-rbac-proxy" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980128 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="4845506f-722e-4a85-abbb-f3bb3801d001" containerName="thanos-querier" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980133 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="config-reloader" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980138 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="99afabe9-bc71-4663-a085-70939405f3dd" containerName="otc-container" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980143 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="prometheus" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980149 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e91d58-487f-44fd-a972-4c872191374d" containerName="alertmanager" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980157 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="method-gate" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980162 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="kube-rbac-proxy" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980167 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="config-reloader" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980172 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7492ab70-7728-4bdf-ae23-9842106016fd" containerName="thanos-sidecar" Oct 30 19:01:18.980444 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.980178 5014 memory_manager.go:365] "RemoveStaleState removing state" podUID="102f1fe2-9102-426c-9c6c-1c63c7e7dad8" containerName="prom-label-proxy" Oct 30 19:01:18.983629 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.983586 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ht9cb/must-gather-jbttc" Oct 30 19:01:18.997887 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:18.997863 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ht9cb\"/\"kube-root-ca.crt\"" Oct 30 19:01:19.009261 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.009239 5014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ht9cb\"/\"openshift-service-ca.crt\"" Oct 30 19:01:19.009580 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.009562 5014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ht9cb\"/\"default-dockercfg-b858q\"" Oct 30 19:01:19.010764 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.010745 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ht9cb/must-gather-jbttc"] Oct 30 19:01:19.087566 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.087545 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d069a445-bcb7-48c3-95ca-444da7036f22-must-gather-output\") pod \"must-gather-jbttc\" (UID: \"d069a445-bcb7-48c3-95ca-444da7036f22\") " pod="openshift-must-gather-ht9cb/must-gather-jbttc" Oct 30 19:01:19.087683 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.087624 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hxb\" (UniqueName: \"kubernetes.io/projected/d069a445-bcb7-48c3-95ca-444da7036f22-kube-api-access-47hxb\") pod \"must-gather-jbttc\" (UID: \"d069a445-bcb7-48c3-95ca-444da7036f22\") " pod="openshift-must-gather-ht9cb/must-gather-jbttc" Oct 30 19:01:19.187934 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.187906 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hxb\" (UniqueName: \"kubernetes.io/projected/d069a445-bcb7-48c3-95ca-444da7036f22-kube-api-access-47hxb\") pod \"must-gather-jbttc\" (UID: \"d069a445-bcb7-48c3-95ca-444da7036f22\") " pod="openshift-must-gather-ht9cb/must-gather-jbttc" Oct 30 19:01:19.188020 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.187956 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d069a445-bcb7-48c3-95ca-444da7036f22-must-gather-output\") pod \"must-gather-jbttc\" (UID: \"d069a445-bcb7-48c3-95ca-444da7036f22\") " pod="openshift-must-gather-ht9cb/must-gather-jbttc" Oct 30 19:01:19.188279 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.188263 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d069a445-bcb7-48c3-95ca-444da7036f22-must-gather-output\") pod \"must-gather-jbttc\" (UID: \"d069a445-bcb7-48c3-95ca-444da7036f22\") " pod="openshift-must-gather-ht9cb/must-gather-jbttc" Oct 30 19:01:19.210992 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.210970 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hxb\" (UniqueName: \"kubernetes.io/projected/d069a445-bcb7-48c3-95ca-444da7036f22-kube-api-access-47hxb\") pod \"must-gather-jbttc\" (UID: \"d069a445-bcb7-48c3-95ca-444da7036f22\") " pod="openshift-must-gather-ht9cb/must-gather-jbttc" Oct 30 19:01:19.311500 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.311482 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ht9cb/must-gather-jbttc" Oct 30 19:01:19.606371 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.606297 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ht9cb/must-gather-jbttc"] Oct 30 19:01:19.610730 ip-10-0-133-62 kubenswrapper[5014]: W1030 19:01:19.610700 5014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd069a445_bcb7_48c3_95ca_444da7036f22.slice/crio-f585308c3587b5980f73be08671fb43fe265565f66b549acf0fd052a8312802b WatchSource:0}: Error finding container f585308c3587b5980f73be08671fb43fe265565f66b549acf0fd052a8312802b: Status 404 returned error can't find the container with id f585308c3587b5980f73be08671fb43fe265565f66b549acf0fd052a8312802b Oct 30 19:01:19.612754 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:19.612734 5014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:01:20.276015 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:20.275979 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ht9cb/must-gather-jbttc" event={"ID":"d069a445-bcb7-48c3-95ca-444da7036f22","Type":"ContainerStarted","Data":"f585308c3587b5980f73be08671fb43fe265565f66b549acf0fd052a8312802b"} Oct 30 19:01:21.281860 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:21.281813 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ht9cb/must-gather-jbttc" event={"ID":"d069a445-bcb7-48c3-95ca-444da7036f22","Type":"ContainerStarted","Data":"9a738e35fa03c8f135b0c183113d5a003189153b14718360358664852ab10b67"} Oct 30 19:01:21.281860 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:21.281865 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ht9cb/must-gather-jbttc" event={"ID":"d069a445-bcb7-48c3-95ca-444da7036f22","Type":"ContainerStarted","Data":"43b2991c726e210c87064213238894eaadd6b2731c0bc6be3945ccfea8b095a0"} Oct 30 19:01:21.325583 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:21.325508 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ht9cb/must-gather-jbttc" podStartSLOduration=2.422197304 podStartE2EDuration="3.32548882s" totalImagesPullingTime="903.291516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:21.32394942 +0000 UTC m=+3894.312289927" watchObservedRunningTime="2026-10-30 19:01:21.32548882 +0000 UTC m=+3894.313829308" Oct 30 19:01:23.352578 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:23.352543 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2pp59_e718a01a-548e-4499-95dc-7cbbd6ad6a12/global-pull-secret-syncer/0.log" Oct 30 19:01:24.016447 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:24.016420 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4znfd_e1e9ace2-7d7b-48c4-a4a8-86d0c2ca22bc/connection-checker/0.log" Oct 30 19:01:24.128309 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:24.128284 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9hqsk_a97de0f2-2248-44e3-bee7-9817777a261d/connection-checker/0.log" Oct 30 19:01:24.250023 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:24.250000 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dh588_38a4b7b8-168f-4525-81be-d43ef28e4c13/connection-checker/0.log" Oct 30 19:01:24.725019 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:24.724406 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wv5gn_167ba867-3860-40d7-9d55-9c81bf49930b/konnectivity-agent/0.log" Oct 30 19:01:24.982699 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:24.982628 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-62.ec2.internal_cea7e9c368b2e2c6cdd7c0cacd3b509e/haproxy/0.log" Oct 30 19:01:27.556724 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:27.556698 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 19:01:27.566280 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:27.558937 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 19:01:31.084282 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:31.084047 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-wb2x4_de9d2339-d6be-4332-921d-ec5717b06699/cluster-monitoring-operator/0.log" Oct 30 19:01:32.242210 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.242175 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvhjk_d504699e-3ab5-4169-9902-5be32fc3abdd/node-exporter/0.log" Oct 30 19:01:32.373333 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.373303 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvhjk_d504699e-3ab5-4169-9902-5be32fc3abdd/kube-rbac-proxy/0.log" Oct 30 19:01:32.491075 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.491044 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvhjk_d504699e-3ab5-4169-9902-5be32fc3abdd/init-textfile/0.log" Oct 30 19:01:32.732818 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.732783 5014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz"] Oct 30 19:01:32.739500 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.739466 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.780615 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.767890 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz"] Oct 30 19:01:32.894955 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.894926 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-proc\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.894955 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.894957 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-podres\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.895159 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.895045 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jcl\" (UniqueName: \"kubernetes.io/projected/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-kube-api-access-x2jcl\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.895159 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.895081 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-sys\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.895232 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.895157 5014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-lib-modules\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.995848 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.995763 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-podres\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.996038 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.995848 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jcl\" (UniqueName: \"kubernetes.io/projected/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-kube-api-access-x2jcl\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.996038 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.995863 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-podres\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.996038 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.995884 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-sys\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.996038 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.995938 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-sys\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.996038 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.995950 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-lib-modules\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.996038 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.996001 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-lib-modules\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.996038 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.996024 5014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-proc\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:32.996451 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:32.996103 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-proc\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:33.019521 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:33.019483 5014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jcl\" (UniqueName: \"kubernetes.io/projected/a2410bd6-6169-4b5b-b4d3-9174ff0bdef1-kube-api-access-x2jcl\") pod \"perf-node-gather-daemonset-mn7tz\" (UID: \"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:33.083177 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:33.083143 5014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:33.395870 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:33.395823 5014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz"] Oct 30 19:01:34.377867 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:34.377825 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" event={"ID":"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1","Type":"ContainerStarted","Data":"c516dee74f63fe2b03ad2e5df813408bbdaed9e5dd823f060f0633ca8b39e6ba"} Oct 30 19:01:34.378063 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:34.377881 5014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" event={"ID":"a2410bd6-6169-4b5b-b4d3-9174ff0bdef1","Type":"ContainerStarted","Data":"8778595e11e22c28abbf7891733150c22e9d3ef7b08a10782ba8d96833e3458d"} Oct 30 19:01:34.378063 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:34.377969 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:34.418415 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:34.418370 5014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" podStartSLOduration=2.418357934 podStartE2EDuration="2.418357934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:34.416895155 +0000 UTC m=+3907.405235643" watchObservedRunningTime="2026-10-30 19:01:34.418357934 +0000 UTC m=+3907.406698435" Oct 30 19:01:34.618497 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:34.618466 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-m5hwn_eb477009-3d76-4f94-a25c-3d1320684c68/prometheus-operator-admission-webhook/0.log" Oct 30 19:01:37.154302 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:37.154220 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-htdwd_f53a361f-64db-41a3-abd1-c04251831048/networking-console-plugin/0.log" Oct 30 19:01:38.168673 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:38.168646 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/1.log" Oct 30 19:01:38.179989 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:38.179962 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5mn6l_29549d27-df3a-48d0-8e7b-85fb883b4481/console-operator/2.log" Oct 30 19:01:39.228034 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:39.228002 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-6r46n_d21419a6-e9b0-46d9-97b8-4484fc94c0dd/download-server/0.log" Oct 30 19:01:40.297938 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:40.297908 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-ggqkn_bb2c63b2-119c-4f2e-a7e5-664bf9e9b999/volume-data-source-validator/0.log" Oct 30 19:01:40.417011 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:40.416986 5014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mn7tz" Oct 30 19:01:42.760556 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:42.760482 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-867jv_b26b3373-a3b2-4a0d-bc89-94bd729b849e/dns/0.log" Oct 30 19:01:42.875322 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:42.875296 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-867jv_b26b3373-a3b2-4a0d-bc89-94bd729b849e/kube-rbac-proxy/0.log" Oct 30 19:01:43.263871 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:43.263841 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-94ttz_f1043899-80a3-47a9-9d1c-0421cd68df00/dns-node-resolver/0.log" Oct 30 19:01:43.332950 ip-10-0-133-62 kubenswrapper[5014]: E1030 19:01:43.332917 5014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9gjwg" containerName="extractor" Oct 30 19:01:44.633767 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:44.633735 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b6577d447-5pww9_afb40c3a-eec0-41f8-b406-b5bc75ac66ac/registry/0.log" Oct 30 19:01:44.741351 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:44.741319 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l67qw_f4572363-4113-4e99-a34a-1b33a4cc08ff/node-ca/0.log" Oct 30 19:01:46.795611 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:46.795565 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c5cf884d7-2k5bz_2b407f39-23cc-432f-a5d7-3a377e4762a5/router/0.log" Oct 30 19:01:47.854908 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:47.854881 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vj5k4_de504e55-5beb-4f9a-a08a-b4f4e1f999d6/serve-healthcheck-canary/0.log" Oct 30 19:01:48.937738 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:48.937704 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-j4bgx_f2070749-ec46-433b-975e-470c734e5fe5/insights-operator/0.log" Oct 30 19:01:48.939976 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:48.939952 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-j4bgx_f2070749-ec46-433b-975e-470c734e5fe5/insights-operator/1.log" Oct 30 19:01:49.437687 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:49.437650 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9gjwg_85b62a96-cc23-4b22-ade6-5066759bbec6/kube-rbac-proxy/0.log" Oct 30 19:01:49.554721 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:49.554696 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9gjwg_85b62a96-cc23-4b22-ade6-5066759bbec6/exporter/0.log" Oct 30 19:01:49.676024 ip-10-0-133-62 kubenswrapper[5014]: I1030 19:01:49.675989 5014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9gjwg_85b62a96-cc23-4b22-ade6-5066759bbec6/extractor/0.log"